Self-Driving Car Engineer Nanodegree

Deep Learning

Project: Traffic Sign Recognition Classifier

The aim of this project is to to classify traffic signs in an appropriate categories. The dataset uses the German Traffic Sign Dataset with 43 distinct categories.

This project implements the classifer using a deep learning techinique, called convolutional neural network (CNN). The project implemenation is divided in the following steps.


Step 1: Dataset Exploration

As stated earlier, the German Traffic Signs dataset is used. The traffic signs and their labels are pickled is a dictionary with 4 key/value pairs.

  • features -> the images pixel values, (width, height, channels)
  • labels -> the label of the traffic sign
  • sizes -> the original width and height of the image, (width, height)
  • coords -> coordinates of a bounding box around the sign in the image, (x1, y1, x2, y2).
In [32]:
# Import useful packages

from datetime import timedelta
from IPython.display import Image
from PIL import Image
from scipy.stats import itemfreq
from sklearn.cross_validation import train_test_split # model_selection
from sklearn.preprocessing import LabelBinarizer
from sklearn.utils import resample
from tqdm import tqdm
from zipfile import ZipFile
import collections
import cv2
import math
import matplotlib.gridspec as gridspec
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import pickle
import tensorflow as tf
import time
import warnings
warnings.filterwarnings("ignore")

Load Traffic Signal Data

Let's first read the pickle file to load training and test features and label data. X_train and X_test refer to features (sign images) while y_train and y_test refer to label (sign name).

In [33]:
# Load pickled data
import pickle

# TODO: fill this in based on where you saved the training and testing data
directory = "./traffic-signs-data"
training_file = directory + "/train.p"
testing_file = directory + "/test.p"

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)
    
X_train, y_train = train['features'], train['labels']
X_test, y_test = test['features'], test['labels']

Now, let's load the data dictionary for sign names and check what does each category refer to.

In [34]:
# Read traffic signal data dictionary
ts_data_dictionary = pd.read_csv("./signnames.csv")
ts_data_dictionary
Out[34]:
ClassId SignName
0 0 Speed limit (20km/h)
1 1 Speed limit (30km/h)
2 2 Speed limit (50km/h)
3 3 Speed limit (60km/h)
4 4 Speed limit (70km/h)
5 5 Speed limit (80km/h)
6 6 End of speed limit (80km/h)
7 7 Speed limit (100km/h)
8 8 Speed limit (120km/h)
9 9 No passing
10 10 No passing for vechiles over 3.5 metric tons
11 11 Right-of-way at the next intersection
12 12 Priority road
13 13 Yield
14 14 Stop
15 15 No vechiles
16 16 Vechiles over 3.5 metric tons prohibited
17 17 No entry
18 18 General caution
19 19 Dangerous curve to the left
20 20 Dangerous curve to the right
21 21 Double curve
22 22 Bumpy road
23 23 Slippery road
24 24 Road narrows on the right
25 25 Road work
26 26 Traffic signals
27 27 Pedestrians
28 28 Children crossing
29 29 Bicycles crossing
30 30 Beware of ice/snow
31 31 Wild animals crossing
32 32 End of all speed and passing limits
33 33 Turn right ahead
34 34 Turn left ahead
35 35 Ahead only
36 36 Go straight or right
37 37 Go straight or left
38 38 Keep right
39 39 Keep left
40 40 Roundabout mandatory
41 41 End of no passing
42 42 End of no passing by vechiles over 3.5 metric ...

Statistics

In [35]:
# number of training examples
n_train = len(X_train)

# number of testing examples
n_test = len(X_test)

# shape of an image
image_shape = (X_train[0].shape)

# Number of classes are in the dataset
n_classes = len(np.unique(y_train))

# image size in pixels
imgsize = X_train[0].shape[1]

print("Number of training examples =", n_train)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)
Number of training examples = 39209
Number of testing examples = 12630
Image data shape = (32, 32, 3)
Number of classes = 43
In [36]:
#  Bar chart of traffic sign labels
label_freq = itemfreq(y_train)

def plot_tf_freq(label):
    """Plot frequency of each traffic sign"""
    y_train = label
    label_freq = itemfreq(y_train)
    
    fig = plt.figure(figsize=(8,4))
    plt.bar(label_freq[:,0], label_freq[:,1], align='center', alpha=0.5)
    plt.xlabel("Traffic Sign Labels")
    plt.ylabel("Frequency")
    plt.title("Frequency of Traffic Sign Labels")
    plt.grid(True)
    
    axes = plt.gca()
    axes.set_xlim([-1,44])
    axes.set_ylim([0,2600])
    plt.gcf().set_size_inches(8,4)
    plt.show()
#     fig.savefig("TrafficSignal_Frequency.png")
In [37]:
plot_tf_freq(y_train)
In [38]:
# Statistics of the data

label_min = np.min(label_freq[:,1])
label_max = np.max(label_freq[:,1])
label_average = n_train/n_classes

# Calcualate # underrepresented label category below certain thresholds
threshold = 500
num_underrepresented_labels = np.sum(label_freq[:,1] <= threshold)

print("Minimum sample size in a category =", label_min)
print("Maximum sample size in a category =", label_max)
print("Average samples in a category =", label_average)
print("# Under-represented samples below 500 thresold =", num_underrepresented_labels)
Minimum sample size in a category = 210
Maximum sample size in a category = 2250
Average samples in a category = 911.8372093023256
# Under-represented samples below 500 thresold = 19

The statistics and plot indicate that the data is unbalanced across different traffic sign classes. In order to make it a balanced class problem, additional dataset is generated in the next sub-section.

Plot sample images from each category

In [39]:
# Grid plot images

rows = n_classes
cols = 5

def gridplot_ts(ts_dict, X_train, filename):
    """Plot sample images from each category"""
    
    fig = plt.figure()
    gs1 = gridspec.GridSpec(rows, cols)
    gs1.update(wspace=0.01, hspace=0.02) # set the spacing between axes. 
    scaling = 4
    plt.figure(figsize=(1*scaling,9*scaling))
    
    for label in np.arange(rows):
        for i in np.arange(cols):
            ax1 = plt.subplot(gs1[label*cols+i])
            ax1.set_xticklabels([])
            ax1.set_yticklabels([])
            ax1.set_aspect('equal')
            idx = np.random.choice(ts_dict[label], 1, replace=False)[0] 
    
            plt.subplot(rows,cols,label*cols+i+1)
            plt.imshow(X_train[idx])
            plt.axis('off')
    
    fig.savefig(filename)
    plt.show()
In [40]:
# Build traffic signal dictionary index to find out 
# which image belong to a sign category

ts_dict_index = {}
for numlabel in np.arange(n_classes):
    ts_dict_index[numlabel] = []
    for i in np.where(y_train==numlabel)[0]:
        ts_dict_index[numlabel].append(i)
In [10]:
gridplot_ts(ts_dict_index, X_train, "original_ts.png") 
<matplotlib.figure.Figure at 0x7f72a71cdd30>

Step 2: Preprocess and Generate New Data

As the data is read, it's time to preprocess the data. Grayscaling and image normalization is performed. In order to overcome the imbalance class problem, additional data is generated from the original training data.

In particular, grayscale is chosen over RGB space as it averages color pixel information from all the color spaces.
Image normalization is also an important step to bring the pixel intensities of different images in the threshold [0.0, 1.0] with mean 0.5. This is particularly useful in implementing the same algorithm to different images and in numerical stablilzation.

In the preprocessing step, images are convered to grayscale and then normalized.

Preprocess images (grayscaling and normalization)

In [41]:
#%% Helper Functions

def grayscale(img):
    """Applies the Grayscale transform
    This will return an image with only one color channel
    To see the returned image as grayscale
    you should call plt.imshow(gray, cmap='gray')"""
    return cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    
    
def normalize_image(img):
    """Normalize the image"""
    a = 0
    b = 1
    greyscale_min = 0
    greyscale_max = 255
    return a + ( ( (img - greyscale_min)*(b - a) )/( greyscale_max - greyscale_min ) )


def img_data_normalization(img_data):
    """Perform greyscaling and normalization of images"""    
    img_data_gray_temp = []
    for i in np.arange(len(img_data)):
        img_data_gray_temp.append(grayscale(img_data[i]))
    
    img_data_gray = np.asarray(img_data_gray_temp)
    img_data_normalized = normalize_image(img_data_gray)
    del img_data_gray_temp, img_data_gray  
    
    return img_data_normalized
In [42]:
# Grayscale and normalize train and test images

X_train_gray = img_data_normalization(X_train)
X_test_gray = img_data_normalization(X_test)
print('Images Preprocessed: Grayscaled and Normalized')
Images Preprocessed: Grayscaled and Normalized

Preprocess labels (convert to one-hot encoded vector)

In [43]:
# Turn labels into numbers and apply One-Hot Encoding

encoder = LabelBinarizer()
encoder.fit(y_train)

train_labels = encoder.transform(y_train).astype(np.float32)
test_labels = encoder.transform(y_test).astype(np.float32)
print('Labels converted to one-hot encoded vector')
Labels converted to one-hot encoded vector

Generate Additional Dataset

The additional data is generated from training dataset by randomly picking images and transforming them using translation, angular and shear transformations.

In [44]:
# Helper function to transform, rotate and shear an image
# Adopted from https://carnd-udacity.atlassian.net/wiki/questions/10322627/project-2-unbalanced-data-generating-additional-data-by-jittering-the-original-image

def transform_image(img,ang_range,shear_range,trans_range):
    '''
    This function transforms images to generate new images.
    The function takes in following arguments,
    1- Image
    2- ang_range: Range of angles for rotation
    3- shear_range: Range of values to apply affine transform to
    4- trans_range: Range of values to apply translations over. 
    
    A Random uniform distribution is used to generate different parameters for transformation
    
    '''
    # Rotation

    ang_rot = np.random.uniform(ang_range)-ang_range/2
    rows,cols,ch = img.shape    
    Rot_M = cv2.getRotationMatrix2D((cols/2,rows/2),ang_rot,1)

    # Translation
    tr_x = trans_range*np.random.uniform()-trans_range/2
    tr_y = trans_range*np.random.uniform()-trans_range/2
    Trans_M = np.float32([[1,0,tr_x],[0,1,tr_y]])

    # Shear
    pts1 = np.float32([[5,5],[20,5],[5,20]])

    pt1 = 5+shear_range*np.random.uniform()-shear_range/2
    pt2 = 20+shear_range*np.random.uniform()-shear_range/2

    pts2 = np.float32([[pt1,5],[pt2,pt1],[5,pt2]])

    shear_M = cv2.getAffineTransform(pts1,pts2)
        
    img = cv2.warpAffine(img,Rot_M,(cols,rows))
    img = cv2.warpAffine(img,Trans_M,(cols,rows))
    img = cv2.warpAffine(img,shear_M,(cols,rows))
    
    return img

Now, it is a time to figure out how many additional samples are to be generated for each class label. From the frequency graph of labels in the above section, class # 2 (Speed limit (50km/h)) has highest frequency of 2250. Instead of picking the maximum frequency as maximum number of samples in each class, the maximum number of sample in each class is drawn from a normal distribution with mean equal to (2250 + delta), where delta is a number of choice. For the exercise delta is chosen as 250.

In [15]:
# Count of new data to be generated in each category 

from scipy.stats import norm
max_data_size = label_max+250
scaling_factor = 25

ts_max_labels = np.rint(norm.rvs(loc = max_data_size, size = n_classes, scale = scaling_factor))
new_data_labels_count = (ts_max_labels - label_freq[:,1])

new_data_mul_factor = (new_data_labels_count/label_freq[:,1])

new_counts = (label_freq[:,1]*new_data_mul_factor)

The frequency for original and generated data is plotted.

In [16]:
# plot count of original and new data

def plot_new_data(label, new_label):
    x = range(len(label))
    
    fig = plt.figure(figsize=(8,4))
    
    p1= plt.bar(x, label, align='center', alpha=0.5)
    p2 = plt.bar(x, new_label, align='center', alpha=0.5, color='r',
             bottom=label)
    plt.xlabel("Traffic Sign Labels")
    plt.ylabel("Frequency")
    plt.title("Frequency of Traffic Sign Labels")
    plt.grid(True)
    plt.legend((p1[0], p2[0]), ('Original Data', 'Generated Data'))
    
    axes = plt.gca()
    axes.set_xlim([-1,44])
    axes.set_ylim([0,3400])
#     plt.gcf().set_size_inches(8,4)
    plt.show()
    
plot_new_data(itemfreq(y_train)[:,1], new_counts.astype(int))    

Now, we know additional number of dataset to be generated in each class. An image is randomly picked and additional data is generated by using translation, angular and shear transformations on an image.

In [17]:
#%% Generate additional data - Updated

angle_range = 15
shear_range = 5
translation_range = 5

X_train_temp = X_train
y_train_temp = y_train

new_X_train = X_train_temp
new_y_train = y_train_temp

newts_dict_index = {} #ts_dict_index
inc_index = len(X_train_temp)
threshold = 0.2

for label in np.arange(rows): #rows
    newts_dict_index[label] = []
    
    dummy_array = np.array([np.zeros(X_train_temp[0].shape)])
    dummy_y = np.array([np.zeros(y_train_temp[0].shape)])
    
    for i in np.arange(new_counts[label]): #new_data_labels_count
        idx = np.random.choice(ts_dict_index[label], 1, replace=False)[0]
        img = X_train_temp[idx]

        newimg = transform_image(img,angle_range,shear_range,translation_range)
        dummy_array = np.concatenate((dummy_array,[newimg]))
        dummy_y = np.concatenate((dummy_y,[label]))
        inc_index = inc_index + 1
        newts_dict_index[label].append(inc_index)

    dummy_array = np.delete(dummy_array,0,0)  
    dummy_y = np.delete(dummy_y,0,0)               
    new_X_train = np.concatenate((new_X_train,dummy_array))
    new_y_train = np.concatenate((new_y_train,dummy_y))   
    
print(">>>>> New Data generation Completed.")         
>>>>> New Data generation Completed.

Randomly seleted samples of additional dataset is plotted below.

In [18]:
# Plot sample generated images
gridplot_ts(newts_dict_index, new_X_train, "new_ts.png") 
<matplotlib.figure.Figure at 0x7f72950948d0>

Generate Training, Validation and Test Datasets

In [45]:
# Convert images to float32 and grayscale and normalize the images

new_X_train = np.array(new_X_train, dtype=np.float32)
new_y_train = np.array(new_y_train, dtype=np.float32)
new_X_train_gray = img_data_normalization(new_X_train)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-45-5e1d5fec6131> in <module>()
      1 # Convert images to float32 and grayscale and normalize the images
      2 
----> 3 new_X_train = np.array(new_X_train, dtype=np.float32)
      4 new_y_train = np.array(new_y_train, dtype=np.float32)
      5 new_X_train_gray = img_data_normalization(new_X_train)

NameError: name 'new_X_train' is not defined
In [46]:
# Get randomized datasets for training, validation and test

train_features, valid_features, train_labels, valid_labels = train_test_split(
    new_X_train_gray,
    new_y_train,
    test_size=0.1,
    random_state=832289)

test_features = X_test_gray
test_labels = y_test

print('Training features and labels randomized and split.')
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-46-bc3892f7bcf6> in <module>()
      2 
      3 train_features, valid_features, train_labels, valid_labels = train_test_split(
----> 4     new_X_train_gray,
      5     new_y_train,
      6     test_size=0.1,

NameError: name 'new_X_train_gray' is not defined
In [21]:
freq_train = itemfreq(train_labels)[:,1]
In [22]:
# Plot Frequencies of Training, Validation and TEst Datasets

freq_train = itemfreq(train_labels)[:,1]
freq_test = itemfreq(test_labels)[:,1]
freq_valid = itemfreq(valid_labels)[:,1]

def plot_freq_data(freq_train, freq_valid, freq_test):
    x = range(len(freq_train))
    
    fig = plt.figure(figsize=(8,4))
    
    p1= plt.bar(x, freq_train, align='center', alpha=0.5)
    p2 = plt.bar(x, freq_valid, align='center', alpha=0.5, color='r',
             bottom=freq_train)
    p3 = plt.bar(x, freq_test, align='center', alpha=0.5, color='g')
    plt.xlabel("Traffic Sign Labels")
    plt.ylabel("Frequency")
    plt.title("Frequency of Traffic Sign Labels")
    plt.grid(True)
    plt.legend((p1[0], p2[0], p3[0]), ('Training Data', 'Validation Data', 'Test Data'))
    
    axes = plt.gca()
    axes.set_xlim([-1,44])
    axes.set_ylim([0,3700])
#     plt.gcf().set_size_inches(8,4)
    plt.show()
    
plot_freq_data(freq_train, freq_valid, freq_test) 
In [23]:
#%% Flattening and One-Hot encoding
imgsize = X_train[0].shape[1]

#Flattening images
train_features = np.reshape(train_features,(len(train_features),imgsize*imgsize)).astype(np.float32)
test_features = np.reshape(test_features,(len(test_features),imgsize*imgsize)).astype(np.float32)
valid_features = np.reshape(valid_features,(len(valid_features),imgsize*imgsize)).astype(np.float32)

encoder = LabelBinarizer()
encoder.fit(train_labels)

train_labels = encoder.transform(train_labels).astype(np.float32)
test_labels = encoder.transform(test_labels).astype(np.float32)
valid_labels = encoder.transform(valid_labels).astype(np.float32)

Write Data to Pickle File

In [24]:
# Dump the generate data to pickle file

pickle_file = './traffic-signs-data/traffic_signal.p'

if not os.path.isfile(pickle_file):
    print('Saving data to pickle file...')
    try:
        with open(pickle_file, 'wb') as pfile:
            pickle.dump(
                {
                    'train_dataset': train_features,
                    'train_labels': train_labels,
                    'valid_dataset': valid_features,
                    'valid_labels': valid_labels,
                    'test_dataset': test_features,
                    'test_labels': test_labels
                },
                pfile, pickle.HIGHEST_PROTOCOL)
    except Exception as e:
        print('Unable to save data to', pickle_file, ':', e)
        raise

print('Data cached in pickle file.')
Data cached in pickle file.

Read Data From Pickle File

In [47]:
# Read the Pickle File

pickle_file = './traffic-signs-data/traffic_signal.p'

with open(pickle_file, 'rb') as f:
  pickle_data = pickle.load(f)
  train_features = pickle_data['train_dataset']
  train_labels = pickle_data['train_labels']
  valid_features = pickle_data['valid_dataset']
  valid_labels = pickle_data['valid_labels']
  test_features = pickle_data['test_dataset']
  test_labels = pickle_data['test_labels']
  del pickle_data  # Free up memory

print('Data and modules loaded.')
Data and modules loaded.

Helper functions to Plot Errors

The helper functions are written to plot image classification errors.

In [48]:
# Plot images
#%% Adopted from the Hvass lab and modified
# https://github.com/Hvass-Labs/TensorFlow-Tutorials/blob/master/02_Convolutional_Neural_Network.ipynb

def plot_images(images, cls_true, rows, cols, cls_pred=None):

    img_shape = (imgsize, imgsize)
    
    # Create figure with 3x3 sub-plots.
    fig, axes = plt.subplots(rows, cols)

    for i, ax in enumerate(axes.flat):
        # Plot image.
        ax.imshow(images[i].reshape(img_shape), cmap='binary')

        # Show true and predicted classes.
        if cls_pred is None:
            xlabel = "True: {0}".format(cls_true[i])
        else:
            xlabel = "True: {0}\n Pred: {1}".format(cls_true[i], cls_pred[i])
    
        # Show the classes as the label on the x-axis.
        ax.set_xlabel(xlabel)
        
        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    fig.tight_layout()
    plt.show()
In [49]:
#%% Plot example errors
#%% Adopted from the Hvass lab and modified
# https://github.com/Hvass-Labs/TensorFlow-Tutorials/blob/master/02_Convolutional_Neural_Network.ipynb

def plot_example_errors(images, cls_true, cls_pred, rows, cols):
    # This function is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # correct is a boolean array whether the predicted class
    # is equal to the true class for each image in the test-set.

    # Negate the boolean array.
    rc = rows*cols
    
    correct = (cls_true == cls_pred)
    incorrect = (correct == False)
    
    # Get the images from the test-set that have been
    # incorrectly classified.
    images = images[incorrect][0:rc]
    
    # Get the predicted classes for those images.
    cls_pred = cls_pred[incorrect][0:rc]

    # Get the true classes for those images.
    cls_true = cls_true[incorrect][0:rc]
    
    # Plot the first 9 images.
    plot_images(images, cls_true, rows, cols, cls_pred)

Step 3: Design and Test a CNN Model Architecture

As the regular neural nets don’t scale well to full images, a convolutional neural network (CNN) is used to extract features and later classify traffic sign images. As the features are be non-linear, a single layer CNN won't capture it. Hence a two-layered architecture as shown in the depiciton figure is used.

The details of the layers are as follows:

  • Layer 1: 64 kernels of (5 x 5) size -> ReLu -> Max Pooling (2 x 2) -> Dropout
  • Layer 2: 128 kernels of (5 x 5) size -> ReLu -> Max Pooling (2 x 2) -> Dropout
  • Fully Connected Layer: 1024 nodes -> ReLu -> Dropout
  • Softmax.

An Adam optimizer is used to minimize the loss function. Dropout is used in training model to avoid overfitting.

In [50]:
# Define Tensorflow placeholders

features = tf.placeholder(tf.float32)
labels = tf.placeholder(tf.float32)
keep_prob = tf.placeholder(tf.float32)
In [51]:
# Define feed dictionaries for train, validation, test data

train_feed_dict = {features: train_features, labels: train_labels, keep_prob: 1}
valid_feed_dict = {features: valid_features, labels: valid_labels, keep_prob: 1}
test_feed_dict = {features: test_features, labels: test_labels, keep_prob: 1}

A wrapper to define CNN is implemented.

In [52]:
#%% Adapted from https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/convolutional_network.py
# Create some wrappers for simplicity
def conv2d(x, W, b, strides=1):
    # Conv2D wrapper, with bias and relu activation
    x = tf.nn.conv2d(x, W, strides=[1, strides, strides, 1], padding='SAME')
    x = tf.nn.bias_add(x, b)
    return tf.nn.relu(x)

def maxpool2d(x, k=2):
    # MaxPool2D wrapper
    return tf.nn.max_pool(x, ksize=[1, k, k, 1], strides=[1, k, k, 1],
                          padding='SAME')

# Create CNN model
def conv_net(x, weights, biases, dropout):
    # Reshape input picture
    x = tf.reshape(x, shape=[-1, imgsize, imgsize, 1])

    # Convolution Layer
    conv1 = conv2d(x, weights['wc1'], biases['bc1'])
    # Max Pooling (down-sampling)
    conv1 = maxpool2d(conv1, k=2)
    conv1 = tf.nn.dropout(conv1, dropout)

    # Convolution Layer
    conv2 = conv2d(conv1, weights['wc2'], biases['bc2'])
    # Max Pooling (down-sampling)
    conv2 = maxpool2d(conv2, k=2)
    conv2 = tf.nn.dropout(conv2, dropout)
    
    # Fully connected layer
    # Reshape conv2 output to fit fully connected layer input
    fc1 = tf.reshape(conv2, [-1, weights['wd1'].get_shape().as_list()[0]])
    fc1 = tf.add(tf.matmul(fc1, weights['wd1']), biases['bd1'])
    fc1 = tf.nn.relu(fc1)
    # Apply Dropout
    fc1 = tf.nn.dropout(fc1, dropout)

    # Output, class prediction
    out = tf.add(tf.matmul(fc1, weights['out']), biases['out'])
    return out
In [53]:
# https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/convolutional_network.py
# Define CNN layer properties

ksize= 5                   # kernel size
layer1_kernels = 64        # Number of kernels in layer 1
layer2_kernels = 128       # Number of kernels in layer 2
layer_fc = 1024            # Number of Neurons in fully connected layer

# constants
meanv = 0.0
stddevv = 0.02

# Store layers weight & bias
weights = {
    # 5x5 conv, 1 input, 32 outputs
    'wc1': tf.Variable(tf.truncated_normal([ksize, ksize, 1, layer1_kernels], mean=meanv, stddev=stddevv, dtype=tf.float32)), # 128
    # 5x5 conv, 32 inputs, 64 outputs
    'wc2': tf.Variable(tf.truncated_normal([ksize, ksize, layer1_kernels, layer2_kernels], mean=meanv, stddev=stddevv, dtype=tf.float32)), # 256
    # fully connected, 8*8*64 inputs, 1024 outputs
    'wd1': tf.Variable(tf.truncated_normal([8*8*layer2_kernels, layer_fc], mean=meanv, stddev=stddevv, dtype=tf.float32)), # 1024
    # 1024 inputs, 10 outputs (class prediction)
    'out': tf.Variable(tf.truncated_normal([layer_fc, n_classes], mean=meanv, stddev=stddevv, dtype=tf.float32))
}

biases = {
    'bc1': tf.Variable(tf.zeros([layer1_kernels])),
    'bc2': tf.Variable(tf.zeros([layer2_kernels])),
    'bd1': tf.Variable(tf.zeros([layer_fc])),
    'out': tf.Variable(tf.zeros([n_classes]))
}

# Construct model
pred = conv_net(features, weights, biases, keep_prob)

#############################
# Linear Model
#weights = tf.Variable(tf.truncated_normal((n_input, n_classes)))
#biases = tf.Variable(tf.zeros(n_classes))
#logits = tf.matmul(features, weights) + biases
#logits = -np.amax(logits)
#pred = tf.nn.softmax(logits)
##############################

# Define loss and optimizer
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(pred, labels)
#cross_entropy = tf.reduce_mean(-tf.reduce_sum(labels * tf.log(tf.clip_by_value(pred,1e-10,1.0)), reduction_indices=[1]))
loss = tf.reduce_mean(cross_entropy)

# Evaluate model
y_pred_cls = tf.argmax(pred,1)
correct_pred = tf.equal(tf.argmax(pred, 1), tf.argmax(labels, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))

softmax_pred = tf.nn.softmax(pred)
softmax_pred_top5 = tf.nn.top_k(softmax_pred, k=5, sorted=True)
In [54]:
# Tensorflow GPU configruation setup

config = tf.ConfigProto()
config.gpu_options.allocator_type = 'BFC'

CNN Training

In [33]:
def model_training(epochs = 2, batch_size = 128, learning_rate = 0.001):
	# time
	start_time = time.time()

	optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)

	# The accuracy measured against the validation set
	validation_accuracy = 0.0
	
	init = tf.global_variables_initializer()

	# Measurements use for graphing loss and accuracy
	log_batch_step = 50
	batches = []
	loss_batch = []
	train_acc_batch = []
	valid_acc_batch = []
	
	#######################################################
	with tf.Session(config = config) as session:
		session.run(init)
		
		batch_count = int(math.ceil(len(train_features)/batch_size))

		for epoch_i in range(epochs):
			print('Epoch {:>2}/{} '.format(epoch_i+1, epochs)+'#'*40)
			# Progress bar
#            batches_pbar = tqdm(range(batch_count), desc='Epoch {:>2}/{}'.format(epoch_i+1, epochs), unit='batches')
			batches_pbar = range(batch_count)

			# The training cycle
			for batch_i in batches_pbar:
				# Get a batch of training features and labels
				batch_index = np.random.choice(len(train_features),batch_size, replace=False)
				batch_features = train_features[batch_index]
				batch_labels = train_labels[batch_index]
			
				# Run optimizer and get loss
				_, lossi = session.run( [optimizer, loss], feed_dict={features: batch_features, labels: batch_labels, keep_prob: 0.5})

				# Log every 50 batches
				if not batch_i % log_batch_step:
					# Calculate Training and Validation accuracy
					training_accuracy = session.run(accuracy, feed_dict={features: batch_features, labels: batch_labels, keep_prob: 1.})
					validation_accuracy = session.run(accuracy, feed_dict=valid_feed_dict)
					print('Batch # = {:>4} : Loss : {:19.16f}, Training accuracy = {:f} : Validation accuracy = {:f}'.format(batch_i, lossi, training_accuracy, validation_accuracy))
					# Log batches
					previous_batch = batches[-1] if batches else 0
					batches.append(log_batch_step + previous_batch)
					loss_batch.append(lossi)
					train_acc_batch.append(training_accuracy)
					valid_acc_batch.append(validation_accuracy)

			# Check accuracy against Validation data
#            train_acc = session.run(accuracy, feed_dict=train_feed_dict)
			validation_accuracy = session.run(accuracy, feed_dict=valid_feed_dict)

	loss_plot = plt.subplot(211)
	loss_plot.set_title('Loss')
	loss_plot.plot(batches, loss_batch, 'g')
	loss_plot.set_xlim([batches[0], batches[-1]])
	acc_plot = plt.subplot(212)
	acc_plot.set_title('Accuracy')
	acc_plot.plot(batches, train_acc_batch, 'r', label='Training Accuracy')
	acc_plot.plot(batches, valid_acc_batch, 'b', label='Validation Accuracy')
	acc_plot.set_ylim([0, 1.0])
	acc_plot.set_xlim([batches[0], batches[-1]])
	acc_plot.legend(loc=4)
	plt.tight_layout()
	plt.show()
    
	print('Training accuracy = {:f} : Validation accuracy = {:f}'.format(training_accuracy, validation_accuracy))

	# end time
	end_time = time.time()
	time_delta = end_time - start_time
	print("Time usage : "+ str(timedelta(seconds=int(round(time_delta)))))        
    
	print('#'*80)

Parameter Tuning

Now, it is time to train a CNN model. The parameters (learning_rate, epoch and batch_size) are tuned to pick the best combination of parameters.

In [35]:
# Train the system 
# learning_rate parameter space 

learning_rate_list = [0.5, 0.1, 0.01, 0.001, 0.0001]
print('>>>>>>>>> Model Training for learning_rate = ', learning_rate_list)

for learning_rate in learning_rate_list:
    print('>>>>>>>>>>>>>>>>>>>>> learning_rate = ', learning_rate)
    model_training(epochs = 1, batch_size = 128, learning_rate = learning_rate)
>>>>>>>>> Model Training for learning_rate =  [0.5, 0.1, 0.01, 0.001, 0.0001]
>>>>>>>>>>>>>>>>>>>>> learning_rate =  0.5
Epoch  1/1 ########################################
Batch # =    0 : Loss :  3.7629628181457520, Training accuracy = 0.054688 : Validation accuracy = 0.021812
Batch # =   50 : Loss :  3.8336503505706787, Training accuracy = 0.039062 : Validation accuracy = 0.023397
Batch # =  100 : Loss :  3.7861852645874023, Training accuracy = 0.031250 : Validation accuracy = 0.023770
Batch # =  150 : Loss :  3.8575949668884277, Training accuracy = 0.023438 : Validation accuracy = 0.022837
Batch # =  200 : Loss :  3.8411154747009277, Training accuracy = 0.023438 : Validation accuracy = 0.023210
Batch # =  250 : Loss :  3.7816100120544434, Training accuracy = 0.039062 : Validation accuracy = 0.023490
Batch # =  300 : Loss :  3.8207461833953857, Training accuracy = 0.000000 : Validation accuracy = 0.023397
Batch # =  350 : Loss :  3.8416342735290527, Training accuracy = 0.023438 : Validation accuracy = 0.022465
Batch # =  400 : Loss :  3.8754019737243652, Training accuracy = 0.023438 : Validation accuracy = 0.023304
Batch # =  450 : Loss :  3.8274354934692383, Training accuracy = 0.007812 : Validation accuracy = 0.023117
Batch # =  500 : Loss :  3.8099789619445801, Training accuracy = 0.007812 : Validation accuracy = 0.026566
Batch # =  550 : Loss :  3.7733862400054932, Training accuracy = 0.015625 : Validation accuracy = 0.021905
Batch # =  600 : Loss :  3.8288476467132568, Training accuracy = 0.015625 : Validation accuracy = 0.024702
Batch # =  650 : Loss :  3.8508672714233398, Training accuracy = 0.039062 : Validation accuracy = 0.023863
Batch # =  700 : Loss :  3.7965259552001953, Training accuracy = 0.023438 : Validation accuracy = 0.022185
Batch # =  750 : Loss :  3.8781199455261230, Training accuracy = 0.023438 : Validation accuracy = 0.021532
Training accuracy = 0.023438 : Validation accuracy = 0.022465
Time usage : 0:00:32
################################################################################
>>>>>>>>>>>>>>>>>>>>> learning_rate =  0.1
Epoch  1/1 ########################################
Batch # =    0 : Loss :  3.7602481842041016, Training accuracy = 0.070312 : Validation accuracy = 0.021812
Batch # =   50 : Loss :  3.7516050338745117, Training accuracy = 0.023438 : Validation accuracy = 0.022558
Batch # =  100 : Loss :  3.7806668281555176, Training accuracy = 0.046875 : Validation accuracy = 0.023024
Batch # =  150 : Loss :  3.7803192138671875, Training accuracy = 0.015625 : Validation accuracy = 0.024515
Batch # =  200 : Loss :  3.7575392723083496, Training accuracy = 0.031250 : Validation accuracy = 0.024888
Batch # =  250 : Loss :  3.7653460502624512, Training accuracy = 0.023438 : Validation accuracy = 0.024515
Batch # =  300 : Loss :  3.7758188247680664, Training accuracy = 0.031250 : Validation accuracy = 0.023397
Batch # =  350 : Loss :  3.8126454353332520, Training accuracy = 0.007812 : Validation accuracy = 0.022837
Batch # =  400 : Loss :  3.7684774398803711, Training accuracy = 0.031250 : Validation accuracy = 0.023304
Batch # =  450 : Loss :  3.7575902938842773, Training accuracy = 0.031250 : Validation accuracy = 0.025913
Batch # =  500 : Loss :  3.7761304378509521, Training accuracy = 0.023438 : Validation accuracy = 0.023397
Batch # =  550 : Loss :  3.7898588180541992, Training accuracy = 0.023438 : Validation accuracy = 0.021999
Batch # =  600 : Loss :  3.7640771865844727, Training accuracy = 0.039062 : Validation accuracy = 0.022465
Batch # =  650 : Loss :  3.7847206592559814, Training accuracy = 0.007812 : Validation accuracy = 0.022558
Batch # =  700 : Loss :  3.7835047245025635, Training accuracy = 0.015625 : Validation accuracy = 0.023304
Batch # =  750 : Loss :  3.7639880180358887, Training accuracy = 0.007812 : Validation accuracy = 0.021066
Training accuracy = 0.007812 : Validation accuracy = 0.023770
Time usage : 0:00:32
################################################################################
>>>>>>>>>>>>>>>>>>>>> learning_rate =  0.01
Epoch  1/1 ########################################
Batch # =    0 : Loss :  3.7632250785827637, Training accuracy = 0.039062 : Validation accuracy = 0.021812
Batch # =   50 : Loss :  3.6925754547119141, Training accuracy = 0.054688 : Validation accuracy = 0.040082
Batch # =  100 : Loss :  3.5171484947204590, Training accuracy = 0.148438 : Validation accuracy = 0.109806
Batch # =  150 : Loss :  3.3755025863647461, Training accuracy = 0.132812 : Validation accuracy = 0.150820
Batch # =  200 : Loss :  3.0634837150573730, Training accuracy = 0.250000 : Validation accuracy = 0.294743
Batch # =  250 : Loss :  2.7781691551208496, Training accuracy = 0.414062 : Validation accuracy = 0.359060
Batch # =  300 : Loss :  2.6341865062713623, Training accuracy = 0.375000 : Validation accuracy = 0.333799
Batch # =  350 : Loss :  2.7110185623168945, Training accuracy = 0.351562 : Validation accuracy = 0.351696
Batch # =  400 : Loss :  2.7140641212463379, Training accuracy = 0.437500 : Validation accuracy = 0.439504
Batch # =  450 : Loss :  2.4801220893859863, Training accuracy = 0.468750 : Validation accuracy = 0.429810
Batch # =  500 : Loss :  2.3503208160400391, Training accuracy = 0.445312 : Validation accuracy = 0.495060
Batch # =  550 : Loss :  2.3018243312835693, Training accuracy = 0.468750 : Validation accuracy = 0.482755
Batch # =  600 : Loss :  2.2837858200073242, Training accuracy = 0.484375 : Validation accuracy = 0.491145
Batch # =  650 : Loss :  2.3251085281372070, Training accuracy = 0.578125 : Validation accuracy = 0.517431
Batch # =  700 : Loss :  2.2837929725646973, Training accuracy = 0.531250 : Validation accuracy = 0.500466
Batch # =  750 : Loss :  2.3856761455535889, Training accuracy = 0.554688 : Validation accuracy = 0.542599
Training accuracy = 0.554688 : Validation accuracy = 0.536726
Time usage : 0:00:33
################################################################################
>>>>>>>>>>>>>>>>>>>>> learning_rate =  0.001
Epoch  1/1 ########################################
Batch # =    0 : Loss :  3.7567687034606934, Training accuracy = 0.046875 : Validation accuracy = 0.023210
Batch # =   50 : Loss :  3.6708631515502930, Training accuracy = 0.070312 : Validation accuracy = 0.064224
Batch # =  100 : Loss :  3.1199061870574951, Training accuracy = 0.273438 : Validation accuracy = 0.224273
Batch # =  150 : Loss :  2.2814595699310303, Training accuracy = 0.453125 : Validation accuracy = 0.435496
Batch # =  200 : Loss :  1.7131057977676392, Training accuracy = 0.671875 : Validation accuracy = 0.572148
Batch # =  250 : Loss :  1.4700338840484619, Training accuracy = 0.734375 : Validation accuracy = 0.671234
Batch # =  300 : Loss :  1.2454280853271484, Training accuracy = 0.781250 : Validation accuracy = 0.726976
Batch # =  350 : Loss :  1.1472924947738647, Training accuracy = 0.796875 : Validation accuracy = 0.777218
Batch # =  400 : Loss :  1.1179615259170532, Training accuracy = 0.820312 : Validation accuracy = 0.791480
Batch # =  450 : Loss :  0.7554442882537842, Training accuracy = 0.867188 : Validation accuracy = 0.830630
Batch # =  500 : Loss :  0.9407036900520325, Training accuracy = 0.859375 : Validation accuracy = 0.837621
Batch # =  550 : Loss :  0.7612855434417725, Training accuracy = 0.859375 : Validation accuracy = 0.855238
Batch # =  600 : Loss :  0.7642142772674561, Training accuracy = 0.906250 : Validation accuracy = 0.873042
Batch # =  650 : Loss :  0.7618005871772766, Training accuracy = 0.890625 : Validation accuracy = 0.884414
Batch # =  700 : Loss :  0.4890145957469940, Training accuracy = 0.953125 : Validation accuracy = 0.892244
Batch # =  750 : Loss :  0.5737851262092590, Training accuracy = 0.945312 : Validation accuracy = 0.908464
Training accuracy = 0.945312 : Validation accuracy = 0.900447
Time usage : 0:00:36
################################################################################
>>>>>>>>>>>>>>>>>>>>> learning_rate =  0.0001
Epoch  1/1 ########################################
Batch # =    0 : Loss :  3.7669949531555176, Training accuracy = 0.007812 : Validation accuracy = 0.026846
Batch # =   50 : Loss :  3.7005858421325684, Training accuracy = 0.023438 : Validation accuracy = 0.032625
Batch # =  100 : Loss :  3.6946508884429932, Training accuracy = 0.070312 : Validation accuracy = 0.061428
Batch # =  150 : Loss :  3.6554188728332520, Training accuracy = 0.015625 : Validation accuracy = 0.059004
Batch # =  200 : Loss :  3.7028613090515137, Training accuracy = 0.062500 : Validation accuracy = 0.097502
Batch # =  250 : Loss :  3.4583749771118164, Training accuracy = 0.140625 : Validation accuracy = 0.147558
Batch # =  300 : Loss :  3.3536705970764160, Training accuracy = 0.203125 : Validation accuracy = 0.203393
Batch # =  350 : Loss :  3.0722732543945312, Training accuracy = 0.210938 : Validation accuracy = 0.250373
Batch # =  400 : Loss :  2.9553990364074707, Training accuracy = 0.312500 : Validation accuracy = 0.305835
Batch # =  450 : Loss :  2.5591669082641602, Training accuracy = 0.359375 : Validation accuracy = 0.340884
Batch # =  500 : Loss :  2.4623200893402100, Training accuracy = 0.351562 : Validation accuracy = 0.373881
Batch # =  550 : Loss :  2.3905444145202637, Training accuracy = 0.406250 : Validation accuracy = 0.415268
Batch # =  600 : Loss :  2.2211637496948242, Training accuracy = 0.546875 : Validation accuracy = 0.443885
Batch # =  650 : Loss :  2.3148334026336670, Training accuracy = 0.468750 : Validation accuracy = 0.447986
Batch # =  700 : Loss :  2.1400442123413086, Training accuracy = 0.492188 : Validation accuracy = 0.489467
Batch # =  750 : Loss :  2.2914819717407227, Training accuracy = 0.523438 : Validation accuracy = 0.492916
Training accuracy = 0.523438 : Validation accuracy = 0.492822
Time usage : 0:00:36
################################################################################
In [36]:
# Train the system 
# learning_rate parameter space 

batch_size_list = [64, 128, 256]
print('>>>>>>>>> Model Training for batch_size = ', batch_size_list)

for batch_size in batch_size_list:
    print('>>>>>>>>>>>>>>>>>>>>> learning_rate = ', batch_size)
    model_training(epochs = 1, batch_size = batch_size, learning_rate = 0.001)
>>>>>>>>> Model Training for batch_size =  [64, 128, 256]
>>>>>>>>>>>>>>>>>>>>> learning_rate =  64
Epoch  1/1 ########################################
Batch # =    0 : Loss :  3.7573873996734619, Training accuracy = 0.062500 : Validation accuracy = 0.025354
Batch # =   50 : Loss :  3.6880292892456055, Training accuracy = 0.031250 : Validation accuracy = 0.034210
Batch # =  100 : Loss :  3.6937537193298340, Training accuracy = 0.062500 : Validation accuracy = 0.092655
Batch # =  150 : Loss :  3.1285498142242432, Training accuracy = 0.265625 : Validation accuracy = 0.216163
Batch # =  200 : Loss :  2.5804395675659180, Training accuracy = 0.359375 : Validation accuracy = 0.347502
Batch # =  250 : Loss :  2.4876332283020020, Training accuracy = 0.421875 : Validation accuracy = 0.473620
Batch # =  300 : Loss :  1.7220023870468140, Training accuracy = 0.656250 : Validation accuracy = 0.551640
Batch # =  350 : Loss :  1.8962922096252441, Training accuracy = 0.531250 : Validation accuracy = 0.589672
Batch # =  400 : Loss :  1.2564043998718262, Training accuracy = 0.703125 : Validation accuracy = 0.645600
Batch # =  450 : Loss :  1.4789010286331177, Training accuracy = 0.718750 : Validation accuracy = 0.686148
Batch # =  500 : Loss :  0.9537362456321716, Training accuracy = 0.859375 : Validation accuracy = 0.736391
Batch # =  550 : Loss :  1.4855182170867920, Training accuracy = 0.656250 : Validation accuracy = 0.746271
Batch # =  600 : Loss :  1.0211431980133057, Training accuracy = 0.843750 : Validation accuracy = 0.775820
Batch # =  650 : Loss :  1.1425535678863525, Training accuracy = 0.750000 : Validation accuracy = 0.796234
Batch # =  700 : Loss :  1.1942393779754639, Training accuracy = 0.781250 : Validation accuracy = 0.813012
Batch # =  750 : Loss :  1.2467815876007080, Training accuracy = 0.734375 : Validation accuracy = 0.813479
Batch # =  800 : Loss :  0.8893184661865234, Training accuracy = 0.921875 : Validation accuracy = 0.846383
Batch # =  850 : Loss :  1.0246385335922241, Training accuracy = 0.828125 : Validation accuracy = 0.847222
Batch # =  900 : Loss :  0.8052483797073364, Training accuracy = 0.937500 : Validation accuracy = 0.854866
Batch # =  950 : Loss :  0.6211720705032349, Training accuracy = 0.921875 : Validation accuracy = 0.861577
Batch # = 1000 : Loss :  0.8602435588836670, Training accuracy = 0.875000 : Validation accuracy = 0.872763
Batch # = 1050 : Loss :  1.0063778162002563, Training accuracy = 0.859375 : Validation accuracy = 0.873135
Batch # = 1100 : Loss :  0.6358349323272705, Training accuracy = 0.812500 : Validation accuracy = 0.897558
Batch # = 1150 : Loss :  0.7320885658264160, Training accuracy = 0.953125 : Validation accuracy = 0.885813
Batch # = 1200 : Loss :  0.7595984935760498, Training accuracy = 0.921875 : Validation accuracy = 0.898024
Batch # = 1250 : Loss :  0.8133047819137573, Training accuracy = 0.921875 : Validation accuracy = 0.907718
Batch # = 1300 : Loss :  0.8822201490402222, Training accuracy = 0.921875 : Validation accuracy = 0.895134
Batch # = 1350 : Loss :  0.6223490834236145, Training accuracy = 0.875000 : Validation accuracy = 0.903430
Batch # = 1400 : Loss :  0.5000456571578979, Training accuracy = 0.937500 : Validation accuracy = 0.909955
Batch # = 1450 : Loss :  0.4963377118110657, Training accuracy = 0.953125 : Validation accuracy = 0.918344
Batch # = 1500 : Loss :  0.4511443078517914, Training accuracy = 0.921875 : Validation accuracy = 0.921420
Training accuracy = 0.921875 : Validation accuracy = 0.914616
Time usage : 0:00:52
################################################################################
>>>>>>>>>>>>>>>>>>>>> learning_rate =  128
Epoch  1/1 ########################################
Batch # =    0 : Loss :  3.7636954784393311, Training accuracy = 0.062500 : Validation accuracy = 0.023210
Batch # =   50 : Loss :  3.7573983669281006, Training accuracy = 0.054688 : Validation accuracy = 0.076529
Batch # =  100 : Loss :  2.9736995697021484, Training accuracy = 0.304688 : Validation accuracy = 0.236764
Batch # =  150 : Loss :  2.2418758869171143, Training accuracy = 0.453125 : Validation accuracy = 0.447520
Batch # =  200 : Loss :  1.6852269172668457, Training accuracy = 0.609375 : Validation accuracy = 0.584825
Batch # =  250 : Loss :  1.5781259536743164, Training accuracy = 0.703125 : Validation accuracy = 0.678971
Batch # =  300 : Loss :  1.0719647407531738, Training accuracy = 0.757812 : Validation accuracy = 0.730891
Batch # =  350 : Loss :  1.1703628301620483, Training accuracy = 0.789062 : Validation accuracy = 0.767524
Batch # =  400 : Loss :  0.8204212188720703, Training accuracy = 0.812500 : Validation accuracy = 0.787472
Batch # =  450 : Loss :  0.9280949831008911, Training accuracy = 0.828125 : Validation accuracy = 0.826156
Batch # =  500 : Loss :  0.7537184953689575, Training accuracy = 0.851562 : Validation accuracy = 0.846290
Batch # =  550 : Loss :  0.9156441092491150, Training accuracy = 0.843750 : Validation accuracy = 0.865958
Batch # =  600 : Loss :  0.7436954975128174, Training accuracy = 0.921875 : Validation accuracy = 0.879008
Batch # =  650 : Loss :  0.5887801647186279, Training accuracy = 0.929688 : Validation accuracy = 0.885719
Batch # =  700 : Loss :  0.5273187160491943, Training accuracy = 0.945312 : Validation accuracy = 0.899515
Batch # =  750 : Loss :  0.5226340889930725, Training accuracy = 0.921875 : Validation accuracy = 0.908184
Training accuracy = 0.921875 : Validation accuracy = 0.903710
Time usage : 0:00:34
################################################################################
>>>>>>>>>>>>>>>>>>>>> learning_rate =  256
Epoch  1/1 ########################################
Batch # =    0 : Loss :  3.7605543136596680, Training accuracy = 0.058594 : Validation accuracy = 0.028430
Batch # =   50 : Loss :  3.4855279922485352, Training accuracy = 0.093750 : Validation accuracy = 0.138236
Batch # =  100 : Loss :  2.3495221138000488, Training accuracy = 0.410156 : Validation accuracy = 0.426641
Batch # =  150 : Loss :  1.6042876243591309, Training accuracy = 0.609375 : Validation accuracy = 0.612416
Batch # =  200 : Loss :  1.3907594680786133, Training accuracy = 0.722656 : Validation accuracy = 0.704418
Batch # =  250 : Loss :  1.1666109561920166, Training accuracy = 0.789062 : Validation accuracy = 0.769482
Batch # =  300 : Loss :  0.9420495033264160, Training accuracy = 0.832031 : Validation accuracy = 0.810962
Batch # =  350 : Loss :  0.8531786799430847, Training accuracy = 0.867188 : Validation accuracy = 0.846103
Training accuracy = 0.867188 : Validation accuracy = 0.856171
Time usage : 0:00:27
################################################################################
In [37]:
# Train the system 
# Epochs parameter space 

epochs_list = [1, 2, 3, 4, 5]
print('>>>>>>>>> Model Training for epochs = ', epochs_list)

for epochs in epochs_list:
    print('>>>>>>>>>>>>>>>>>>>>> Epoch = ', epochs)
    model_training(epochs = epochs, batch_size = 128, learning_rate = 0.001)
>>>>>>>>> Model Training for epochs =  [1, 2, 3, 4, 5]
>>>>>>>>>>>>>>>>>>>>> Epoch =  1
Epoch  1/1 ########################################
Batch # =    0 : Loss :  3.7630639076232910, Training accuracy = 0.062500 : Validation accuracy = 0.023117
Batch # =   50 : Loss :  3.6531376838684082, Training accuracy = 0.062500 : Validation accuracy = 0.069911
Batch # =  100 : Loss :  2.8657159805297852, Training accuracy = 0.335938 : Validation accuracy = 0.247390
Batch # =  150 : Loss :  2.3123655319213867, Training accuracy = 0.468750 : Validation accuracy = 0.462808
Batch # =  200 : Loss :  1.5883324146270752, Training accuracy = 0.632812 : Validation accuracy = 0.580816
Batch # =  250 : Loss :  1.6284472942352295, Training accuracy = 0.632812 : Validation accuracy = 0.677666
Batch # =  300 : Loss :  1.3450977802276611, Training accuracy = 0.789062 : Validation accuracy = 0.740212
Batch # =  350 : Loss :  0.9473516941070557, Training accuracy = 0.867188 : Validation accuracy = 0.766499
Batch # =  400 : Loss :  0.8446941971778870, Training accuracy = 0.828125 : Validation accuracy = 0.799217
Batch # =  450 : Loss :  0.9180850982666016, Training accuracy = 0.859375 : Validation accuracy = 0.836782
Batch # =  500 : Loss :  0.6457986831665039, Training accuracy = 0.929688 : Validation accuracy = 0.841163
Batch # =  550 : Loss :  0.6941310167312622, Training accuracy = 0.882812 : Validation accuracy = 0.856450
Batch # =  600 : Loss :  0.7329275608062744, Training accuracy = 0.914062 : Validation accuracy = 0.876025
Batch # =  650 : Loss :  0.5961716175079346, Training accuracy = 0.953125 : Validation accuracy = 0.891312
Batch # =  700 : Loss :  0.5431079268455505, Training accuracy = 0.953125 : Validation accuracy = 0.899235
Batch # =  750 : Loss :  0.8024932742118835, Training accuracy = 0.898438 : Validation accuracy = 0.901100
Training accuracy = 0.898438 : Validation accuracy = 0.906040
Time usage : 0:00:35
################################################################################
>>>>>>>>>>>>>>>>>>>>> Epoch =  2
Epoch  1/2 ########################################
Batch # =    0 : Loss :  3.7646884918212891, Training accuracy = 0.078125 : Validation accuracy = 0.026473
Batch # =   50 : Loss :  3.6164779663085938, Training accuracy = 0.117188 : Validation accuracy = 0.064970
Batch # =  100 : Loss :  3.0705714225769043, Training accuracy = 0.242188 : Validation accuracy = 0.234433
Batch # =  150 : Loss :  2.3484354019165039, Training accuracy = 0.500000 : Validation accuracy = 0.452461
Batch # =  200 : Loss :  1.7471573352813721, Training accuracy = 0.593750 : Validation accuracy = 0.593866
Batch # =  250 : Loss :  1.5600535869598389, Training accuracy = 0.679688 : Validation accuracy = 0.668624
Batch # =  300 : Loss :  1.0163383483886719, Training accuracy = 0.796875 : Validation accuracy = 0.721383
Batch # =  350 : Loss :  1.2755688428878784, Training accuracy = 0.765625 : Validation accuracy = 0.771346
Batch # =  400 : Loss :  0.8311594724655151, Training accuracy = 0.835938 : Validation accuracy = 0.813106
Batch # =  450 : Loss :  0.9705199003219604, Training accuracy = 0.843750 : Validation accuracy = 0.834825
Batch # =  500 : Loss :  0.7295638322830200, Training accuracy = 0.906250 : Validation accuracy = 0.852442
Batch # =  550 : Loss :  0.7208256721496582, Training accuracy = 0.906250 : Validation accuracy = 0.864467
Batch # =  600 : Loss :  0.5542557239532471, Training accuracy = 0.914062 : Validation accuracy = 0.884694
Batch # =  650 : Loss :  0.6137109994888306, Training accuracy = 0.890625 : Validation accuracy = 0.883855
Batch # =  700 : Loss :  0.6617394685745239, Training accuracy = 0.906250 : Validation accuracy = 0.898210
Batch # =  750 : Loss :  0.6170150041580200, Training accuracy = 0.929688 : Validation accuracy = 0.911446
Epoch  2/2 ########################################
Batch # =    0 : Loss :  0.4584920704364777, Training accuracy = 0.937500 : Validation accuracy = 0.907625
Batch # =   50 : Loss :  0.5433520674705505, Training accuracy = 0.898438 : Validation accuracy = 0.916014
Batch # =  100 : Loss :  0.4617624282836914, Training accuracy = 0.960938 : Validation accuracy = 0.925522
Batch # =  150 : Loss :  0.5159948468208313, Training accuracy = 0.968750 : Validation accuracy = 0.927759
Batch # =  200 : Loss :  0.6292016506195068, Training accuracy = 0.921875 : Validation accuracy = 0.932233
Batch # =  250 : Loss :  0.5155491232872009, Training accuracy = 0.953125 : Validation accuracy = 0.934657
Batch # =  300 : Loss :  0.3564810156822205, Training accuracy = 0.945312 : Validation accuracy = 0.938012
Batch # =  350 : Loss :  0.3968178927898407, Training accuracy = 0.953125 : Validation accuracy = 0.940809
Batch # =  400 : Loss :  0.5004026293754578, Training accuracy = 0.968750 : Validation accuracy = 0.951528
Batch # =  450 : Loss :  0.3694377541542053, Training accuracy = 0.953125 : Validation accuracy = 0.948639
Batch # =  500 : Loss :  0.3492299914360046, Training accuracy = 0.992188 : Validation accuracy = 0.949012
Batch # =  550 : Loss :  0.3287321329116821, Training accuracy = 0.968750 : Validation accuracy = 0.951994
Batch # =  600 : Loss :  0.2867155075073242, Training accuracy = 0.968750 : Validation accuracy = 0.955350
Batch # =  650 : Loss :  0.3041206896305084, Training accuracy = 0.976562 : Validation accuracy = 0.958333
Batch # =  700 : Loss :  0.3620144426822662, Training accuracy = 0.945312 : Validation accuracy = 0.959079
Batch # =  750 : Loss :  0.2341283857822418, Training accuracy = 0.976562 : Validation accuracy = 0.962248
Training accuracy = 0.976562 : Validation accuracy = 0.964578
Time usage : 0:01:09
################################################################################
>>>>>>>>>>>>>>>>>>>>> Epoch =  3
Epoch  1/3 ########################################
Batch # =    0 : Loss :  3.7633516788482666, Training accuracy = 0.062500 : Validation accuracy = 0.020134
Batch # =   50 : Loss :  3.6805174350738525, Training accuracy = 0.085938 : Validation accuracy = 0.068512
Batch # =  100 : Loss :  2.7845282554626465, Training accuracy = 0.265625 : Validation accuracy = 0.253356
Batch # =  150 : Loss :  2.1112856864929199, Training accuracy = 0.492188 : Validation accuracy = 0.433165
Batch # =  200 : Loss :  2.0299892425537109, Training accuracy = 0.546875 : Validation accuracy = 0.570097
Batch # =  250 : Loss :  1.3647859096527100, Training accuracy = 0.703125 : Validation accuracy = 0.662099
Batch # =  300 : Loss :  1.1448287963867188, Training accuracy = 0.757812 : Validation accuracy = 0.712808
Batch # =  350 : Loss :  1.2141633033752441, Training accuracy = 0.820312 : Validation accuracy = 0.765194
Batch # =  400 : Loss :  1.1384235620498657, Training accuracy = 0.796875 : Validation accuracy = 0.795022
Batch # =  450 : Loss :  0.8113938570022583, Training accuracy = 0.875000 : Validation accuracy = 0.828020
Batch # =  500 : Loss :  0.9201327562332153, Training accuracy = 0.835938 : Validation accuracy = 0.842934
Batch # =  550 : Loss :  0.8065974712371826, Training accuracy = 0.890625 : Validation accuracy = 0.863068
Batch # =  600 : Loss :  0.7189948558807373, Training accuracy = 0.867188 : Validation accuracy = 0.883855
Batch # =  650 : Loss :  0.6761445999145508, Training accuracy = 0.890625 : Validation accuracy = 0.895880
Batch # =  700 : Loss :  0.5340215563774109, Training accuracy = 0.921875 : Validation accuracy = 0.888609
Batch # =  750 : Loss :  0.6055451631546021, Training accuracy = 0.937500 : Validation accuracy = 0.905667
Epoch  2/3 ########################################
Batch # =    0 : Loss :  0.6424761414527893, Training accuracy = 0.937500 : Validation accuracy = 0.913124
Batch # =   50 : Loss :  0.4214668869972229, Training accuracy = 0.945312 : Validation accuracy = 0.912192
Batch # =  100 : Loss :  0.4166619777679443, Training accuracy = 0.937500 : Validation accuracy = 0.916480
Batch # =  150 : Loss :  0.4416112005710602, Training accuracy = 0.968750 : Validation accuracy = 0.928784
Batch # =  200 : Loss :  0.4090186059474945, Training accuracy = 0.945312 : Validation accuracy = 0.928318
Batch # =  250 : Loss :  0.3392913341522217, Training accuracy = 0.992188 : Validation accuracy = 0.938292
Batch # =  300 : Loss :  0.4511552155017853, Training accuracy = 0.937500 : Validation accuracy = 0.935402
Batch # =  350 : Loss :  0.2882480025291443, Training accuracy = 0.984375 : Validation accuracy = 0.939690
Batch # =  400 : Loss :  0.4455330371856689, Training accuracy = 0.960938 : Validation accuracy = 0.945283
Batch # =  450 : Loss :  0.3240980505943298, Training accuracy = 0.937500 : Validation accuracy = 0.944444
Batch # =  500 : Loss :  0.2263068556785583, Training accuracy = 1.000000 : Validation accuracy = 0.949478
Batch # =  550 : Loss :  0.4462352395057678, Training accuracy = 0.937500 : Validation accuracy = 0.948546
Batch # =  600 : Loss :  0.4318864643573761, Training accuracy = 0.968750 : Validation accuracy = 0.954232
Batch # =  650 : Loss :  0.3570327758789062, Training accuracy = 0.960938 : Validation accuracy = 0.954884
Batch # =  700 : Loss :  0.2503566443920135, Training accuracy = 0.984375 : Validation accuracy = 0.957028
Batch # =  750 : Loss :  0.2675922811031342, Training accuracy = 0.953125 : Validation accuracy = 0.955257
Epoch  3/3 ########################################
Batch # =    0 : Loss :  0.3864265084266663, Training accuracy = 0.976562 : Validation accuracy = 0.955630
Batch # =   50 : Loss :  0.2977392971515656, Training accuracy = 0.968750 : Validation accuracy = 0.958333
Batch # =  100 : Loss :  0.4189139604568481, Training accuracy = 0.960938 : Validation accuracy = 0.962434
Batch # =  150 : Loss :  0.1960378140211105, Training accuracy = 0.968750 : Validation accuracy = 0.959452
Batch # =  200 : Loss :  0.2211703807115555, Training accuracy = 0.992188 : Validation accuracy = 0.958706
Batch # =  250 : Loss :  0.3138988614082336, Training accuracy = 0.984375 : Validation accuracy = 0.959172
Batch # =  300 : Loss :  0.2416050583124161, Training accuracy = 0.984375 : Validation accuracy = 0.963460
Batch # =  350 : Loss :  0.2744901776313782, Training accuracy = 0.984375 : Validation accuracy = 0.965417
Batch # =  400 : Loss :  0.3261872529983521, Training accuracy = 0.984375 : Validation accuracy = 0.969425
Batch # =  450 : Loss :  0.2338826507329941, Training accuracy = 0.992188 : Validation accuracy = 0.968307
Batch # =  500 : Loss :  0.2960625290870667, Training accuracy = 0.984375 : Validation accuracy = 0.967841
Batch # =  550 : Loss :  0.3180631399154663, Training accuracy = 0.976562 : Validation accuracy = 0.967654
Batch # =  600 : Loss :  0.2000383883714676, Training accuracy = 0.992188 : Validation accuracy = 0.971103
Batch # =  650 : Loss :  0.3438125550746918, Training accuracy = 0.968750 : Validation accuracy = 0.970637
Batch # =  700 : Loss :  0.2385281175374985, Training accuracy = 0.968750 : Validation accuracy = 0.976044
Batch # =  750 : Loss :  0.1623654514551163, Training accuracy = 1.000000 : Validation accuracy = 0.973620
Training accuracy = 1.000000 : Validation accuracy = 0.972875
Time usage : 0:01:44
################################################################################
>>>>>>>>>>>>>>>>>>>>> Epoch =  4
Epoch  1/4 ########################################
Batch # =    0 : Loss :  3.7596797943115234, Training accuracy = 0.078125 : Validation accuracy = 0.025168
Batch # =   50 : Loss :  3.6780729293823242, Training accuracy = 0.101562 : Validation accuracy = 0.085477
Batch # =  100 : Loss :  2.8606054782867432, Training accuracy = 0.289062 : Validation accuracy = 0.287286
Batch # =  150 : Loss :  1.9081075191497803, Training accuracy = 0.523438 : Validation accuracy = 0.485925
Batch # =  200 : Loss :  1.6881783008575439, Training accuracy = 0.687500 : Validation accuracy = 0.597595
Batch # =  250 : Loss :  1.6059601306915283, Training accuracy = 0.640625 : Validation accuracy = 0.687360
Batch # =  300 : Loss :  1.0818175077438354, Training accuracy = 0.812500 : Validation accuracy = 0.717002
Batch # =  350 : Loss :  1.1298377513885498, Training accuracy = 0.781250 : Validation accuracy = 0.760906
Batch # =  400 : Loss :  0.9076474308967590, Training accuracy = 0.843750 : Validation accuracy = 0.810403
Batch # =  450 : Loss :  1.1476974487304688, Training accuracy = 0.843750 : Validation accuracy = 0.832494
Batch # =  500 : Loss :  0.8857166767120361, Training accuracy = 0.828125 : Validation accuracy = 0.850578
Batch # =  550 : Loss :  0.5688501000404358, Training accuracy = 0.937500 : Validation accuracy = 0.867916
Batch # =  600 : Loss :  0.7296909689903259, Training accuracy = 0.859375 : Validation accuracy = 0.876025
Batch # =  650 : Loss :  0.6713007092475891, Training accuracy = 0.929688 : Validation accuracy = 0.894202
Batch # =  700 : Loss :  0.6621425151824951, Training accuracy = 0.937500 : Validation accuracy = 0.898210
Batch # =  750 : Loss :  0.4146552085876465, Training accuracy = 0.960938 : Validation accuracy = 0.899888
Epoch  2/4 ########################################
Batch # =    0 : Loss :  0.5728905200958252, Training accuracy = 0.960938 : Validation accuracy = 0.910701
Batch # =   50 : Loss :  0.4104195237159729, Training accuracy = 0.953125 : Validation accuracy = 0.919649
Batch # =  100 : Loss :  0.4250195324420929, Training accuracy = 0.984375 : Validation accuracy = 0.918997
Batch # =  150 : Loss :  0.4576641917228699, Training accuracy = 0.945312 : Validation accuracy = 0.925056
Batch # =  200 : Loss :  0.3849958777427673, Training accuracy = 0.968750 : Validation accuracy = 0.930835
Batch # =  250 : Loss :  0.4486990869045258, Training accuracy = 0.914062 : Validation accuracy = 0.935123
Batch # =  300 : Loss :  0.3950422704219818, Training accuracy = 0.937500 : Validation accuracy = 0.940902
Batch # =  350 : Loss :  0.3948450386524200, Training accuracy = 0.960938 : Validation accuracy = 0.945656
Batch # =  400 : Loss :  0.6266871094703674, Training accuracy = 0.945312 : Validation accuracy = 0.945749
Batch # =  450 : Loss :  0.4345409274101257, Training accuracy = 0.984375 : Validation accuracy = 0.948080
Batch # =  500 : Loss :  0.3885608911514282, Training accuracy = 0.968750 : Validation accuracy = 0.947707
Batch # =  550 : Loss :  0.4063389301300049, Training accuracy = 0.976562 : Validation accuracy = 0.950317
Batch # =  600 : Loss :  0.3291890621185303, Training accuracy = 0.984375 : Validation accuracy = 0.952088
Batch # =  650 : Loss :  0.3524245023727417, Training accuracy = 0.953125 : Validation accuracy = 0.955071
Batch # =  700 : Loss :  0.2256165593862534, Training accuracy = 0.976562 : Validation accuracy = 0.957774
Batch # =  750 : Loss :  0.2104183882474899, Training accuracy = 0.992188 : Validation accuracy = 0.957960
Epoch  3/4 ########################################
Batch # =    0 : Loss :  0.3608540296554565, Training accuracy = 0.953125 : Validation accuracy = 0.961502
Batch # =   50 : Loss :  0.3345354795455933, Training accuracy = 0.953125 : Validation accuracy = 0.959918
Batch # =  100 : Loss :  0.4540509581565857, Training accuracy = 0.976562 : Validation accuracy = 0.959172
Batch # =  150 : Loss :  0.2315764725208282, Training accuracy = 0.984375 : Validation accuracy = 0.959918
Batch # =  200 : Loss :  0.2656944394111633, Training accuracy = 0.984375 : Validation accuracy = 0.965044
Batch # =  250 : Loss :  0.2582076191902161, Training accuracy = 0.976562 : Validation accuracy = 0.965697
Batch # =  300 : Loss :  0.2045233249664307, Training accuracy = 0.976562 : Validation accuracy = 0.967934
Batch # =  350 : Loss :  0.2874214947223663, Training accuracy = 0.984375 : Validation accuracy = 0.965231
Batch # =  400 : Loss :  0.2870500683784485, Training accuracy = 0.984375 : Validation accuracy = 0.969892
Batch # =  450 : Loss :  0.3175325989723206, Training accuracy = 0.992188 : Validation accuracy = 0.969612
Batch # =  500 : Loss :  0.4056153893470764, Training accuracy = 0.953125 : Validation accuracy = 0.972781
Batch # =  550 : Loss :  0.2935949265956879, Training accuracy = 0.976562 : Validation accuracy = 0.969239
Batch # =  600 : Loss :  0.3259557187557220, Training accuracy = 0.992188 : Validation accuracy = 0.970544
Batch # =  650 : Loss :  0.1957689821720123, Training accuracy = 0.992188 : Validation accuracy = 0.971569
Batch # =  700 : Loss :  0.2456584423780441, Training accuracy = 0.984375 : Validation accuracy = 0.974925
Batch # =  750 : Loss :  0.1610826849937439, Training accuracy = 1.000000 : Validation accuracy = 0.973713
Epoch  4/4 ########################################
Batch # =    0 : Loss :  0.1678114086389542, Training accuracy = 0.976562 : Validation accuracy = 0.970171
Batch # =   50 : Loss :  0.2336588501930237, Training accuracy = 0.968750 : Validation accuracy = 0.972129
Batch # =  100 : Loss :  0.2624144256114960, Training accuracy = 1.000000 : Validation accuracy = 0.971476
Batch # =  150 : Loss :  0.3076999187469482, Training accuracy = 0.968750 : Validation accuracy = 0.972502
Batch # =  200 : Loss :  0.2755848169326782, Training accuracy = 0.984375 : Validation accuracy = 0.977442
Batch # =  250 : Loss :  0.2030195593833923, Training accuracy = 0.984375 : Validation accuracy = 0.978001
Batch # =  300 : Loss :  0.2693772912025452, Training accuracy = 0.992188 : Validation accuracy = 0.976789
Batch # =  350 : Loss :  0.1601667106151581, Training accuracy = 0.992188 : Validation accuracy = 0.978374
Batch # =  400 : Loss :  0.2503364682197571, Training accuracy = 0.984375 : Validation accuracy = 0.979400
Batch # =  450 : Loss :  0.2300987392663956, Training accuracy = 0.992188 : Validation accuracy = 0.977628
Batch # =  500 : Loss :  0.2306560873985291, Training accuracy = 0.968750 : Validation accuracy = 0.977815
Batch # =  550 : Loss :  0.2250596433877945, Training accuracy = 1.000000 : Validation accuracy = 0.979865
Batch # =  600 : Loss :  0.1446179449558258, Training accuracy = 1.000000 : Validation accuracy = 0.982103
Batch # =  650 : Loss :  0.1597056090831757, Training accuracy = 0.992188 : Validation accuracy = 0.979120
Batch # =  700 : Loss :  0.1706656664609909, Training accuracy = 0.992188 : Validation accuracy = 0.980704
Batch # =  750 : Loss :  0.2208470702171326, Training accuracy = 0.992188 : Validation accuracy = 0.980425
Training accuracy = 0.992188 : Validation accuracy = 0.979586
Time usage : 0:02:17
################################################################################
>>>>>>>>>>>>>>>>>>>>> Epoch =  5
Epoch  1/5 ########################################
Batch # =    0 : Loss :  3.7564482688903809, Training accuracy = 0.054688 : Validation accuracy = 0.021812
Batch # =   50 : Loss :  3.6174235343933105, Training accuracy = 0.109375 : Validation accuracy = 0.083240
Batch # =  100 : Loss :  2.9220466613769531, Training accuracy = 0.265625 : Validation accuracy = 0.238069
Batch # =  150 : Loss :  2.4198336601257324, Training accuracy = 0.429688 : Validation accuracy = 0.431208
Batch # =  200 : Loss :  1.8930081129074097, Training accuracy = 0.578125 : Validation accuracy = 0.558538
Batch # =  250 : Loss :  1.4974833726882935, Training accuracy = 0.671875 : Validation accuracy = 0.643177
Batch # =  300 : Loss :  1.3527483940124512, Training accuracy = 0.703125 : Validation accuracy = 0.721383
Batch # =  350 : Loss :  1.3141744136810303, Training accuracy = 0.765625 : Validation accuracy = 0.752423
Batch # =  400 : Loss :  0.8166924715042114, Training accuracy = 0.851562 : Validation accuracy = 0.795022
Batch # =  450 : Loss :  0.9894106984138489, Training accuracy = 0.820312 : Validation accuracy = 0.818512
Batch # =  500 : Loss :  0.9450118541717529, Training accuracy = 0.835938 : Validation accuracy = 0.841443
Batch # =  550 : Loss :  0.7767740488052368, Training accuracy = 0.828125 : Validation accuracy = 0.863068
Batch # =  600 : Loss :  0.7256450057029724, Training accuracy = 0.906250 : Validation accuracy = 0.874441
Batch # =  650 : Loss :  0.7372870445251465, Training accuracy = 0.898438 : Validation accuracy = 0.889728
Batch # =  700 : Loss :  0.4873059391975403, Training accuracy = 0.937500 : Validation accuracy = 0.901379
Batch # =  750 : Loss :  0.6364665031433105, Training accuracy = 0.914062 : Validation accuracy = 0.909302
Epoch  2/5 ########################################
Batch # =    0 : Loss :  0.5840971469879150, Training accuracy = 0.953125 : Validation accuracy = 0.904176
Batch # =   50 : Loss :  0.5947526097297668, Training accuracy = 0.921875 : Validation accuracy = 0.912565
Batch # =  100 : Loss :  0.5070885419845581, Training accuracy = 0.937500 : Validation accuracy = 0.915827
Batch # =  150 : Loss :  0.4272981882095337, Training accuracy = 0.953125 : Validation accuracy = 0.928691
Batch # =  200 : Loss :  0.4117142260074615, Training accuracy = 0.976562 : Validation accuracy = 0.929250
Batch # =  250 : Loss :  0.4874212145805359, Training accuracy = 0.945312 : Validation accuracy = 0.936428
Batch # =  300 : Loss :  0.5604411959648132, Training accuracy = 0.976562 : Validation accuracy = 0.936707
Batch # =  350 : Loss :  0.3600858449935913, Training accuracy = 0.984375 : Validation accuracy = 0.933259
Batch # =  400 : Loss :  0.5369583368301392, Training accuracy = 0.953125 : Validation accuracy = 0.946308
Batch # =  450 : Loss :  0.3893875479698181, Training accuracy = 0.960938 : Validation accuracy = 0.951062
Batch # =  500 : Loss :  0.3313043713569641, Training accuracy = 0.937500 : Validation accuracy = 0.954977
Batch # =  550 : Loss :  0.3187344074249268, Training accuracy = 0.960938 : Validation accuracy = 0.952647
Batch # =  600 : Loss :  0.2628998756408691, Training accuracy = 1.000000 : Validation accuracy = 0.953206
Batch # =  650 : Loss :  0.3228351473808289, Training accuracy = 0.953125 : Validation accuracy = 0.957960
Batch # =  700 : Loss :  0.2905387878417969, Training accuracy = 0.984375 : Validation accuracy = 0.960104
Batch # =  750 : Loss :  0.2358021140098572, Training accuracy = 0.976562 : Validation accuracy = 0.961502
Epoch  3/5 ########################################
Batch # =    0 : Loss :  0.4430100917816162, Training accuracy = 0.968750 : Validation accuracy = 0.959918
Batch # =   50 : Loss :  0.4284247756004333, Training accuracy = 0.960938 : Validation accuracy = 0.960757
Batch # =  100 : Loss :  0.4384738504886627, Training accuracy = 0.968750 : Validation accuracy = 0.960850
Batch # =  150 : Loss :  0.2952984869480133, Training accuracy = 0.976562 : Validation accuracy = 0.963553
Batch # =  200 : Loss :  0.1801398992538452, Training accuracy = 0.992188 : Validation accuracy = 0.964392
Batch # =  250 : Loss :  0.3074157238006592, Training accuracy = 1.000000 : Validation accuracy = 0.967561
Batch # =  300 : Loss :  0.3064900040626526, Training accuracy = 0.984375 : Validation accuracy = 0.967002
Batch # =  350 : Loss :  0.1813340336084366, Training accuracy = 0.976562 : Validation accuracy = 0.967934
Batch # =  400 : Loss :  0.2223556637763977, Training accuracy = 0.976562 : Validation accuracy = 0.970637
Batch # =  450 : Loss :  0.2713360190391541, Training accuracy = 0.984375 : Validation accuracy = 0.971569
Batch # =  500 : Loss :  0.3492352962493896, Training accuracy = 0.984375 : Validation accuracy = 0.971383
Batch # =  550 : Loss :  0.2024696916341782, Training accuracy = 0.984375 : Validation accuracy = 0.972781
Batch # =  600 : Loss :  0.1874496042728424, Training accuracy = 0.992188 : Validation accuracy = 0.971290
Batch # =  650 : Loss :  0.2640672028064728, Training accuracy = 0.968750 : Validation accuracy = 0.973247
Batch # =  700 : Loss :  0.4018089175224304, Training accuracy = 0.976562 : Validation accuracy = 0.974646
Batch # =  750 : Loss :  0.1404107809066772, Training accuracy = 0.992188 : Validation accuracy = 0.975484
Epoch  4/5 ########################################
Batch # =    0 : Loss :  0.2177401036024094, Training accuracy = 0.992188 : Validation accuracy = 0.974179
Batch # =   50 : Loss :  0.1599374413490295, Training accuracy = 0.992188 : Validation accuracy = 0.973900
Batch # =  100 : Loss :  0.2366738319396973, Training accuracy = 1.000000 : Validation accuracy = 0.972968
Batch # =  150 : Loss :  0.2101553231477737, Training accuracy = 0.992188 : Validation accuracy = 0.976417
Batch # =  200 : Loss :  0.3126921951770782, Training accuracy = 0.968750 : Validation accuracy = 0.974925
Batch # =  250 : Loss :  0.1516934186220169, Training accuracy = 0.984375 : Validation accuracy = 0.974925
Batch # =  300 : Loss :  0.1562338620424271, Training accuracy = 0.984375 : Validation accuracy = 0.979213
Batch # =  350 : Loss :  0.3249799311161041, Training accuracy = 0.992188 : Validation accuracy = 0.976696
Batch # =  400 : Loss :  0.2099163532257080, Training accuracy = 0.992188 : Validation accuracy = 0.979679
Batch # =  450 : Loss :  0.2056630551815033, Training accuracy = 0.984375 : Validation accuracy = 0.978188
Batch # =  500 : Loss :  0.1620208919048309, Training accuracy = 0.992188 : Validation accuracy = 0.979306
Batch # =  550 : Loss :  0.1500347256660461, Training accuracy = 1.000000 : Validation accuracy = 0.980052
Batch # =  600 : Loss :  0.1587107181549072, Training accuracy = 0.992188 : Validation accuracy = 0.981077
Batch # =  650 : Loss :  0.2881190776824951, Training accuracy = 0.984375 : Validation accuracy = 0.977908
Batch # =  700 : Loss :  0.1747491657733917, Training accuracy = 0.992188 : Validation accuracy = 0.979399
Batch # =  750 : Loss :  0.2223739475011826, Training accuracy = 0.992188 : Validation accuracy = 0.981543
Epoch  5/5 ########################################
Batch # =    0 : Loss :  0.1904412209987640, Training accuracy = 0.984375 : Validation accuracy = 0.982196
Batch # =   50 : Loss :  0.1607957631349564, Training accuracy = 0.976562 : Validation accuracy = 0.982103
Batch # =  100 : Loss :  0.1650978326797485, Training accuracy = 1.000000 : Validation accuracy = 0.984433
Batch # =  150 : Loss :  0.2673680484294891, Training accuracy = 0.984375 : Validation accuracy = 0.983408
Batch # =  200 : Loss :  0.1616698503494263, Training accuracy = 0.992188 : Validation accuracy = 0.981543
Batch # =  250 : Loss :  0.1884258985519409, Training accuracy = 1.000000 : Validation accuracy = 0.982941
Batch # =  300 : Loss :  0.1318645924329758, Training accuracy = 0.992188 : Validation accuracy = 0.981450
Batch # =  350 : Loss :  0.0884827226400375, Training accuracy = 0.992188 : Validation accuracy = 0.983594
Batch # =  400 : Loss :  0.0843067839741707, Training accuracy = 1.000000 : Validation accuracy = 0.983314
Batch # =  450 : Loss :  0.1398147493600845, Training accuracy = 1.000000 : Validation accuracy = 0.983967
Batch # =  500 : Loss :  0.1863041371107101, Training accuracy = 1.000000 : Validation accuracy = 0.983594
Batch # =  550 : Loss :  0.1445603668689728, Training accuracy = 0.992188 : Validation accuracy = 0.983035
Batch # =  600 : Loss :  0.1807740926742554, Training accuracy = 0.992188 : Validation accuracy = 0.984526
Batch # =  650 : Loss :  0.0982081517577171, Training accuracy = 1.000000 : Validation accuracy = 0.982848
Batch # =  700 : Loss :  0.1748105734586716, Training accuracy = 0.984375 : Validation accuracy = 0.985924
Batch # =  750 : Loss :  0.2131069302558899, Training accuracy = 0.984375 : Validation accuracy = 0.984713
Training accuracy = 0.984375 : Validation accuracy = 0.984433
Time usage : 0:02:49
################################################################################

Based on the above studies, I selected the following parameters for testing sets to avoid overfitting.

  • epochs = 4
  • batch_size = 128
  • learning_rate = 0.001.

The model is trained with the selected parameters and validation accuracy is calculated.

In [38]:
model_training(epochs = 4, batch_size = 128, learning_rate = 0.001)
Epoch  1/4 ########################################
Batch # =    0 : Loss :  3.7584764957427979, Training accuracy = 0.054688 : Validation accuracy = 0.024888
Batch # =   50 : Loss :  3.6336197853088379, Training accuracy = 0.078125 : Validation accuracy = 0.059284
Batch # =  100 : Loss :  3.0265471935272217, Training accuracy = 0.234375 : Validation accuracy = 0.247670
Batch # =  150 : Loss :  2.1023225784301758, Training accuracy = 0.468750 : Validation accuracy = 0.455444
Batch # =  200 : Loss :  1.8535778522491455, Training accuracy = 0.554688 : Validation accuracy = 0.592748
Batch # =  250 : Loss :  1.4646797180175781, Training accuracy = 0.695312 : Validation accuracy = 0.672725
Batch # =  300 : Loss :  1.4651470184326172, Training accuracy = 0.742188 : Validation accuracy = 0.719798
Batch # =  350 : Loss :  1.2098798751831055, Training accuracy = 0.765625 : Validation accuracy = 0.747483
Batch # =  400 : Loss :  0.9477650523185730, Training accuracy = 0.859375 : Validation accuracy = 0.792785
Batch # =  450 : Loss :  0.7029190063476562, Training accuracy = 0.890625 : Validation accuracy = 0.815995
Batch # =  500 : Loss :  0.9236096143722534, Training accuracy = 0.843750 : Validation accuracy = 0.851976
Batch # =  550 : Loss :  0.6609702110290527, Training accuracy = 0.906250 : Validation accuracy = 0.843121
Batch # =  600 : Loss :  0.8410717844963074, Training accuracy = 0.906250 : Validation accuracy = 0.872763
Batch # =  650 : Loss :  0.6762938499450684, Training accuracy = 0.929688 : Validation accuracy = 0.888329
Batch # =  700 : Loss :  0.5239769816398621, Training accuracy = 0.929688 : Validation accuracy = 0.899422
Batch # =  750 : Loss :  0.6114540696144104, Training accuracy = 0.937500 : Validation accuracy = 0.899888
Epoch  2/4 ########################################
Batch # =    0 : Loss :  0.7232260704040527, Training accuracy = 0.914062 : Validation accuracy = 0.907158
Batch # =   50 : Loss :  0.5480903387069702, Training accuracy = 0.929688 : Validation accuracy = 0.907718
Batch # =  100 : Loss :  0.3998004794120789, Training accuracy = 0.968750 : Validation accuracy = 0.918251
Batch # =  150 : Loss :  0.5037077069282532, Training accuracy = 0.953125 : Validation accuracy = 0.924590
Batch # =  200 : Loss :  0.3774853944778442, Training accuracy = 0.968750 : Validation accuracy = 0.934843
Batch # =  250 : Loss :  0.5724378824234009, Training accuracy = 0.976562 : Validation accuracy = 0.936521
Batch # =  300 : Loss :  0.5312235355377197, Training accuracy = 0.968750 : Validation accuracy = 0.941275
Batch # =  350 : Loss :  0.4429323077201843, Training accuracy = 0.945312 : Validation accuracy = 0.943326
Batch # =  400 : Loss :  0.3417262434959412, Training accuracy = 0.945312 : Validation accuracy = 0.944910
Batch # =  450 : Loss :  0.5464233160018921, Training accuracy = 0.945312 : Validation accuracy = 0.946961
Batch # =  500 : Loss :  0.3448261022567749, Training accuracy = 0.960938 : Validation accuracy = 0.950596
Batch # =  550 : Loss :  0.2853217720985413, Training accuracy = 0.976562 : Validation accuracy = 0.951622
Batch # =  600 : Loss :  0.2919290661811829, Training accuracy = 0.976562 : Validation accuracy = 0.953020
Batch # =  650 : Loss :  0.3283066451549530, Training accuracy = 0.976562 : Validation accuracy = 0.954698
Batch # =  700 : Loss :  0.4823163747787476, Training accuracy = 0.976562 : Validation accuracy = 0.959918
Batch # =  750 : Loss :  0.4103860557079315, Training accuracy = 0.960938 : Validation accuracy = 0.959731
Epoch  3/4 ########################################
Batch # =    0 : Loss :  0.3842298984527588, Training accuracy = 0.960938 : Validation accuracy = 0.959452
Batch # =   50 : Loss :  0.2479669004678726, Training accuracy = 0.992188 : Validation accuracy = 0.963553
Batch # =  100 : Loss :  0.2416794747114182, Training accuracy = 0.992188 : Validation accuracy = 0.966443
Batch # =  150 : Loss :  0.2443135976791382, Training accuracy = 0.984375 : Validation accuracy = 0.968027
Batch # =  200 : Loss :  0.3575897216796875, Training accuracy = 0.992188 : Validation accuracy = 0.963367
Batch # =  250 : Loss :  0.2471392303705215, Training accuracy = 0.992188 : Validation accuracy = 0.965417
Batch # =  300 : Loss :  0.2874321043491364, Training accuracy = 0.984375 : Validation accuracy = 0.965883
Batch # =  350 : Loss :  0.2435759007930756, Training accuracy = 0.976562 : Validation accuracy = 0.967468
Batch # =  400 : Loss :  0.2286189943552017, Training accuracy = 0.984375 : Validation accuracy = 0.969519
Batch # =  450 : Loss :  0.3155519068241119, Training accuracy = 0.976562 : Validation accuracy = 0.972502
Batch # =  500 : Loss :  0.1552656888961792, Training accuracy = 0.992188 : Validation accuracy = 0.972502
Batch # =  550 : Loss :  0.3112837672233582, Training accuracy = 0.992188 : Validation accuracy = 0.969892
Batch # =  600 : Loss :  0.1385972499847412, Training accuracy = 0.984375 : Validation accuracy = 0.972315
Batch # =  650 : Loss :  0.3016411662101746, Training accuracy = 0.968750 : Validation accuracy = 0.975391
Batch # =  700 : Loss :  0.1496367156505585, Training accuracy = 1.000000 : Validation accuracy = 0.973434
Batch # =  750 : Loss :  0.3339200615882874, Training accuracy = 0.992188 : Validation accuracy = 0.974832
Epoch  4/4 ########################################
Batch # =    0 : Loss :  0.2271589636802673, Training accuracy = 0.992188 : Validation accuracy = 0.974739
Batch # =   50 : Loss :  0.1723051071166992, Training accuracy = 1.000000 : Validation accuracy = 0.974366
Batch # =  100 : Loss :  0.1744600236415863, Training accuracy = 0.992188 : Validation accuracy = 0.974552
Batch # =  150 : Loss :  0.2045464813709259, Training accuracy = 1.000000 : Validation accuracy = 0.978561
Batch # =  200 : Loss :  0.1664565652608871, Training accuracy = 1.000000 : Validation accuracy = 0.977628
Batch # =  250 : Loss :  0.3196319341659546, Training accuracy = 0.984375 : Validation accuracy = 0.978281
Batch # =  300 : Loss :  0.3323807120323181, Training accuracy = 0.976562 : Validation accuracy = 0.978840
Batch # =  350 : Loss :  0.1368412971496582, Training accuracy = 0.984375 : Validation accuracy = 0.979306
Batch # =  400 : Loss :  0.2795813679695129, Training accuracy = 0.976562 : Validation accuracy = 0.978094
Batch # =  450 : Loss :  0.1318466365337372, Training accuracy = 0.992188 : Validation accuracy = 0.980145
Batch # =  500 : Loss :  0.1296583265066147, Training accuracy = 0.992188 : Validation accuracy = 0.980332
Batch # =  550 : Loss :  0.1718593537807465, Training accuracy = 0.976562 : Validation accuracy = 0.981357
Batch # =  600 : Loss :  0.2076384425163269, Training accuracy = 0.984375 : Validation accuracy = 0.978281
Batch # =  650 : Loss :  0.1051831841468811, Training accuracy = 1.000000 : Validation accuracy = 0.983035
Batch # =  700 : Loss :  0.3261499106884003, Training accuracy = 0.984375 : Validation accuracy = 0.980798
Batch # =  750 : Loss :  0.2085514664649963, Training accuracy = 0.992188 : Validation accuracy = 0.982848
Training accuracy = 0.992188 : Validation accuracy = 0.982382
Time usage : 0:02:18
################################################################################

The final training accuracy is ~ 99% and the validation accuracy is ~98%.

CNN Model Testing on Test Dataset

In [15]:
def model_testing(epochs = 4, batch_size = 128, learning_rate = 0.001):
	
	# time
	start_time = time.time()

	optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)
	# The accuracy measured against the test set
	test_accuracy = 0.0
	
	init = tf.global_variables_initializer()

	# Measurements use for graphing loss and accuracy
	log_batch_step = 50
	batches = []
	loss_batch = []
	train_acc_batch = []
	test_acc_batch = []
	
	#######################################################
	with tf.Session(config = config) as session:
		session.run(init)
		
		batch_count = int(math.ceil(len(train_features)/batch_size))

		for epoch_i in range(epochs):
			print('Epoch {:>2}/{} '.format(epoch_i+1, epochs)+'#'*40)
			# Progress bar
			batches_pbar = range(batch_count)

			# The training cycle
			for batch_i in batches_pbar:
				# Get a batch of training features and labels
				batch_index = np.random.choice(len(train_features),batch_size, replace=False)
				batch_features = train_features[batch_index]
				batch_labels = train_labels[batch_index]
			
				# Run optimizer and get loss
				_, lossi = session.run( [optimizer, loss], feed_dict={features: batch_features, labels: batch_labels, keep_prob: 0.5})

				# Log every 50 batches
				if not batch_i % log_batch_step:
					# Calculate Training and Validation accuracy
					training_accuracy = session.run(accuracy, feed_dict={features: batch_features, labels: batch_labels, keep_prob: 1.})
					test_accuracy = session.run(accuracy, feed_dict=test_feed_dict)
					print('Batch # = {:>4} : Loss : {:19.16f}, Training accuracy = {:f} : Test accuracy = {:f}'.format(batch_i, lossi, training_accuracy, test_accuracy))
					# Log batches
					previous_batch = batches[-1] if batches else 0
					batches.append(log_batch_step + previous_batch)
					loss_batch.append(lossi)
					train_acc_batch.append(training_accuracy)
					test_acc_batch.append(test_accuracy)

			# Check accuracy against Validation data
			test_accuracy = session.run(accuracy, feed_dict=test_feed_dict)           
			y_pred = session.run(y_pred_cls, feed_dict=test_feed_dict)
			
			cm = tf.contrib.metrics.confusion_matrix(tf.argmax(test_labels, 1), y_pred, num_classes=n_classes, dtype=tf.int32).eval()
			
	loss_plot = plt.subplot(211)
	loss_plot.set_title('Loss')
	loss_plot.plot(batches, loss_batch, 'g')
	loss_plot.set_xlim([batches[0], batches[-1]])
	acc_plot = plt.subplot(212)
	acc_plot.set_title('Accuracy')
	acc_plot.plot(batches, train_acc_batch, 'r', label='Training Accuracy')
	acc_plot.plot(batches, test_acc_batch, 'b', label='Test Accuracy')
	acc_plot.set_ylim([0, 1.0])
	acc_plot.set_xlim([batches[0], batches[-1]])
	acc_plot.legend(loc=4)
	plt.tight_layout()
	plt.show()

	print('>>>>> Test accuracy = {:f}'.format(test_accuracy))
	print('Confusion Matrix : \n', cm)

	# end time
	end_time = time.time()
	time_delta = end_time - start_time
	print("Time usage : "+ str(timedelta(seconds=int(round(time_delta))))) 
	print('#'*80)
    
	return test_accuracy, y_pred, cm
In [16]:
# Testing the model
print('>>>>>>>>> Model Testing ')
test_accuracy, y_pred, cm = model_testing(epochs = 4, batch_size = 128, learning_rate = 0.001)
>>>>>>>>> Model Testing 
Epoch  1/4 ########################################
Batch # =    0 : Loss :  3.7592422962188721, Training accuracy = 0.046875 : Test accuracy = 0.009501
Batch # =   50 : Loss :  3.6580352783203125, Training accuracy = 0.078125 : Test accuracy = 0.048852
Batch # =  100 : Loss :  3.1677501201629639, Training accuracy = 0.210938 : Test accuracy = 0.188044
Batch # =  150 : Loss :  2.3099513053894043, Training accuracy = 0.476562 : Test accuracy = 0.455820
Batch # =  200 : Loss :  1.7073459625244141, Training accuracy = 0.625000 : Test accuracy = 0.557245
Batch # =  250 : Loss :  1.5524237155914307, Training accuracy = 0.695312 : Test accuracy = 0.678227
Batch # =  300 : Loss :  1.7116169929504395, Training accuracy = 0.656250 : Test accuracy = 0.726366
Batch # =  350 : Loss :  1.2153507471084595, Training accuracy = 0.765625 : Test accuracy = 0.784482
Batch # =  400 : Loss :  0.8968414068222046, Training accuracy = 0.820312 : Test accuracy = 0.790341
Batch # =  450 : Loss :  1.0804164409637451, Training accuracy = 0.835938 : Test accuracy = 0.830008
Batch # =  500 : Loss :  1.1204254627227783, Training accuracy = 0.867188 : Test accuracy = 0.846398
Batch # =  550 : Loss :  0.8303377628326416, Training accuracy = 0.921875 : Test accuracy = 0.856137
Batch # =  600 : Loss :  0.7484161853790283, Training accuracy = 0.851562 : Test accuracy = 0.851940
Batch # =  650 : Loss :  0.7863209247589111, Training accuracy = 0.882812 : Test accuracy = 0.870863
Batch # =  700 : Loss :  0.6338762044906616, Training accuracy = 0.945312 : Test accuracy = 0.868726
Batch # =  750 : Loss :  0.7559843063354492, Training accuracy = 0.906250 : Test accuracy = 0.883136
Epoch  2/4 ########################################
Batch # =    0 : Loss :  0.6188810467720032, Training accuracy = 0.921875 : Test accuracy = 0.880365
Batch # =   50 : Loss :  0.7447621822357178, Training accuracy = 0.875000 : Test accuracy = 0.892716
Batch # =  100 : Loss :  0.7772140502929688, Training accuracy = 0.898438 : Test accuracy = 0.904909
Batch # =  150 : Loss :  0.4059754014015198, Training accuracy = 0.921875 : Test accuracy = 0.899130
Batch # =  200 : Loss :  0.5442161560058594, Training accuracy = 0.921875 : Test accuracy = 0.910135
Batch # =  250 : Loss :  0.5575675964355469, Training accuracy = 0.929688 : Test accuracy = 0.903643
Batch # =  300 : Loss :  0.4237042963504791, Training accuracy = 0.953125 : Test accuracy = 0.902851
Batch # =  350 : Loss :  0.5152720212936401, Training accuracy = 0.960938 : Test accuracy = 0.910293
Batch # =  400 : Loss :  0.4152316153049469, Training accuracy = 0.953125 : Test accuracy = 0.909977
Batch # =  450 : Loss :  0.3892438113689423, Training accuracy = 0.976562 : Test accuracy = 0.916153
Batch # =  500 : Loss :  0.3661605119705200, Training accuracy = 0.953125 : Test accuracy = 0.916153
Batch # =  550 : Loss :  0.5381246209144592, Training accuracy = 0.945312 : Test accuracy = 0.915994
Batch # =  600 : Loss :  0.4150263369083405, Training accuracy = 0.945312 : Test accuracy = 0.915044
Batch # =  650 : Loss :  0.4066990017890930, Training accuracy = 0.960938 : Test accuracy = 0.924387
Batch # =  700 : Loss :  0.3569338023662567, Training accuracy = 0.968750 : Test accuracy = 0.922170
Batch # =  750 : Loss :  0.3930897116661072, Training accuracy = 0.960938 : Test accuracy = 0.933017
Epoch  3/4 ########################################
Batch # =    0 : Loss :  0.4150699973106384, Training accuracy = 0.968750 : Test accuracy = 0.929613
Batch # =   50 : Loss :  0.3827723264694214, Training accuracy = 0.953125 : Test accuracy = 0.928425
Batch # =  100 : Loss :  0.2462710291147232, Training accuracy = 0.984375 : Test accuracy = 0.926525
Batch # =  150 : Loss :  0.2254491746425629, Training accuracy = 0.984375 : Test accuracy = 0.925575
Batch # =  200 : Loss :  0.2833667397499084, Training accuracy = 0.976562 : Test accuracy = 0.935155
Batch # =  250 : Loss :  0.2489475905895233, Training accuracy = 0.968750 : Test accuracy = 0.932700
Batch # =  300 : Loss :  0.3766182065010071, Training accuracy = 0.984375 : Test accuracy = 0.929454
Batch # =  350 : Loss :  0.2196498513221741, Training accuracy = 0.992188 : Test accuracy = 0.922328
Batch # =  400 : Loss :  0.3078953027725220, Training accuracy = 0.992188 : Test accuracy = 0.936580
Batch # =  450 : Loss :  0.2962912321090698, Training accuracy = 0.984375 : Test accuracy = 0.933017
Batch # =  500 : Loss :  0.1980132609605789, Training accuracy = 0.984375 : Test accuracy = 0.933888
Batch # =  550 : Loss :  0.2004681676626205, Training accuracy = 0.976562 : Test accuracy = 0.929771
Batch # =  600 : Loss :  0.2974828481674194, Training accuracy = 0.992188 : Test accuracy = 0.939193
Batch # =  650 : Loss :  0.2083019167184830, Training accuracy = 0.976562 : Test accuracy = 0.940776
Batch # =  700 : Loss :  0.2357169687747955, Training accuracy = 1.000000 : Test accuracy = 0.943627
Batch # =  750 : Loss :  0.3711652755737305, Training accuracy = 0.984375 : Test accuracy = 0.935076
Epoch  4/4 ########################################
Batch # =    0 : Loss :  0.2215653210878372, Training accuracy = 0.960938 : Test accuracy = 0.933017
Batch # =   50 : Loss :  0.1566378921270370, Training accuracy = 0.992188 : Test accuracy = 0.934284
Batch # =  100 : Loss :  0.3070029616355896, Training accuracy = 0.968750 : Test accuracy = 0.940935
Batch # =  150 : Loss :  0.2558541893959045, Training accuracy = 0.976562 : Test accuracy = 0.939430
Batch # =  200 : Loss :  0.2427286207675934, Training accuracy = 0.984375 : Test accuracy = 0.931750
Batch # =  250 : Loss :  0.1654802411794662, Training accuracy = 1.000000 : Test accuracy = 0.936738
Batch # =  300 : Loss :  0.1422621905803680, Training accuracy = 0.984375 : Test accuracy = 0.941172
Batch # =  350 : Loss :  0.2648428082466125, Training accuracy = 0.992188 : Test accuracy = 0.941806
Batch # =  400 : Loss :  0.3591000437736511, Training accuracy = 0.976562 : Test accuracy = 0.940856
Batch # =  450 : Loss :  0.2880754470825195, Training accuracy = 0.984375 : Test accuracy = 0.941172
Batch # =  500 : Loss :  0.2199274301528931, Training accuracy = 1.000000 : Test accuracy = 0.936184
Batch # =  550 : Loss :  0.1862096935510635, Training accuracy = 0.992188 : Test accuracy = 0.944260
Batch # =  600 : Loss :  0.1930680274963379, Training accuracy = 1.000000 : Test accuracy = 0.947190
Batch # =  650 : Loss :  0.2188165783882141, Training accuracy = 1.000000 : Test accuracy = 0.942914
Batch # =  700 : Loss :  0.3148793578147888, Training accuracy = 0.992188 : Test accuracy = 0.944577
Batch # =  750 : Loss :  0.1675575673580170, Training accuracy = 0.992188 : Test accuracy = 0.947665
>>>>> Test accuracy = 0.944735
Confusion Matrix : 
 [[ 52   8   0 ...,   0   0   0]
 [ 10 699   3 ...,   0   0   0]
 [  0  13 700 ...,   0   0   0]
 ..., 
 [  0   3   0 ...,  68   0   0]
 [  0   0   0 ...,   0  60   0]
 [  0   0   0 ...,   0   0  90]]
Time usage : 0:02:30
################################################################################

A test accuracy of ~94% is achieved.

Now let's plot some of the mis-classified images.

In [17]:
# Plot example errors in testing
images = test_features
 
cls_true = test_labels
cls_true = np.argmax(cls_true, 1)
cls_pred = y_pred

rows = 4
cols = 4

plot_example_errors(images, cls_true, cls_pred, rows, cols)

Step 4: Testing CNN Model on New Images

A new data is genererated by extracting traffic signal images from Gram Dataset and Google Images. Nine images are saved in a directory and read. A following procedure is used to create a simple processing pipeline:

  • Images are read and resized to 32 x 32 pixel size
  • Images are convered to Grayscale
  • Images are then normalized
  • Images are reshaped and labels are convered to one-hot encoded vector for further analysis.
In [55]:
#%% Read new dataset

# Iterate through the list of images
imagelist = ["newts/"+imgname for imgname in os.listdir("newts/")]

newts_features = np.array([np.zeros((imgsize,imgsize))])

for image in imagelist:
    if image.endswith(".png"):
        
        # Read and preprocess the image 
        image_original = mpimg.imread(image)
        img = (np.copy(image_original)*255).astype('uint8')
        croppedImage = cv2.resize(img, (imgsize, imgsize))
        gray = grayscale(croppedImage)
        nimg = normalize_image(gray)
        newts_features = np.concatenate((newts_features,[nimg]))
        
newts_features = np.delete(newts_features,0,0)
newts_labels = np.array([31, 20, 27, 40, 25, 0, 1, 14, 28])

newts_features = np.reshape(newts_features,(len(newts_features),imgsize*imgsize)).astype(np.float32)
newts_labels = encoder.transform(newts_labels).astype(np.float32)
print('Labels converted to one-hot encoded vector')
Labels converted to one-hot encoded vector
Image Correct Label Comment Prediction Estimate
31 Wild animal crossing. Similar images occurs in the database. Easy
20 Dangerous curve to the right. Similar images occurs in the database. Easy
27 This is a pedestrian image taken from pixelated traffic signal. The person is moving right instead of left in the database. Due to pixelation, this is a difficult image to classify. Difficult
40 Roundabout mandatory. Similar images occurs in the database. Easy
25 Road work. This image has yellow background in contrast to white one in the database. Easy
0 This is a purposely miscalssified image. No such images exist in the database. Difficult
1 Speed limit (30km/h). Similar images occurs in the database. Easy
14 Stop. The shadow on the image may make it difficult to classify Difficult
28 Children crossing. The image in the database is slightly different. This may make the classification difficult. Difficult

Here are the gray colored plots of images with their true labels.

In [56]:
#%% plot images - new test dataset
rows = 3
cols = rows
rc = rows*cols

images = newts_features

cls_true = newts_labels
cls_true = np.argmax(cls_true, 1)

plot_images(images, cls_true, rows, cols)
In [57]:
# Feed dictionary
newts_feed_dict = {features: newts_features, labels: newts_labels, keep_prob: 1}
In [77]:
def model_testing_newdata(epochs = 4, batch_size = 128, learning_rate = 0.001):

	start_time = time.time()
	optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)
	init = tf.global_variables_initializer()

	# Measurements use for graphing loss and accuracy
	log_batch_step = 50
	batches = []
	loss_batch = []
	train_acc_batch = []
	test_acc_batch = []
	#######################################################
	with tf.Session(config = config) as session:
		session.run(init)
		
		batch_count = int(math.ceil(len(train_features)/batch_size))

		for epoch_i in range(epochs):
			print('Epoch {:>2}/{} '.format(epoch_i+1, epochs)+'#'*40)
			# Progress bar
			batches_pbar = range(batch_count)

			# The training cycle
			for batch_i in batches_pbar:
				# Get a batch of training features and labels
				batch_index = np.random.choice(len(train_features),batch_size, replace=False)
				batch_features = train_features[batch_index]
				batch_labels = train_labels[batch_index]
			
				# Run optimizer and get loss
				_, lossi = session.run( [optimizer, loss], feed_dict={features: batch_features, labels: batch_labels, keep_prob: 0.5})
				
				# Log every 50 batches
				if not batch_i % log_batch_step:
					# Calculate Training and Validation accuracy
					training_accuracy = session.run(accuracy, feed_dict={features: batch_features, labels: batch_labels, keep_prob: 1.})
					test_accuracy = session.run(accuracy, feed_dict=newts_feed_dict)
					print('Batch # = {:>4} : Loss : {:19.16f}, Training accuracy = {:f} : Test accuracy = {:f}'.format(batch_i, lossi, training_accuracy, test_accuracy))
					# Log batches
					previous_batch = batches[-1] if batches else 0
					batches.append(log_batch_step + previous_batch)
					loss_batch.append(lossi)
					train_acc_batch.append(training_accuracy)
					test_acc_batch.append(test_accuracy)
                    
			# Check accuracy against Validation data
			newts_accurary = session.run(accuracy, feed_dict=newts_feed_dict)
			y_pred_newts = session.run(y_pred_cls, feed_dict=newts_feed_dict)
			softmax_newts = session.run(softmax_pred, feed_dict=newts_feed_dict)
			softmax_pred_top5_newts = session.run(softmax_pred_top5, feed_dict=newts_feed_dict)

	loss_plot = plt.subplot(211)
	loss_plot.set_title('Loss')
	loss_plot.plot(batches, loss_batch, 'g')
	loss_plot.set_xlim([batches[0], batches[-1]])
	acc_plot = plt.subplot(212)
	acc_plot.set_title('Accuracy')
	acc_plot.plot(batches, train_acc_batch, 'r', label='Training Accuracy')
	acc_plot.plot(batches, test_acc_batch, 'b', label='Test Accuracy')
	acc_plot.set_ylim([0, 1.0])
	acc_plot.set_xlim([batches[0], batches[-1]])
	acc_plot.legend(loc=4)
	plt.tight_layout()
	plt.show()
    
	print('Training accuracy = {:f} : Validation accuracy = {:f}'.format(training_accuracy, test_accuracy))
    
	# end time
	end_time = time.time()
	time_delta = end_time - start_time
	print("Time usage : "+ str(timedelta(seconds=int(round(time_delta)))))    
    
	return newts_accurary, y_pred_newts, softmax_newts, softmax_pred_top5_newts
In [79]:
# Testing the model on new dataset
print('>>>>>>>>> Model Testing on New Dataset')
newts_accurary, y_pred_newts, softmax_newts, softmax_pred_top5_newts = model_testing_newdata(epochs = 3, batch_size = 128, learning_rate = 0.001)
>>>>>>>>> Model Testing on New Dataset
Epoch  1/3 ########################################
Batch # =    0 : Loss :  3.7587478160858154, Training accuracy = 0.054688 : Test accuracy = 0.111111
Batch # =   50 : Loss :  3.6195914745330811, Training accuracy = 0.101562 : Test accuracy = 0.000000
Batch # =  100 : Loss :  2.9087052345275879, Training accuracy = 0.273438 : Test accuracy = 0.111111
Batch # =  150 : Loss :  2.5577759742736816, Training accuracy = 0.406250 : Test accuracy = 0.222222
Batch # =  200 : Loss :  1.9896135330200195, Training accuracy = 0.570312 : Test accuracy = 0.222222
Batch # =  250 : Loss :  1.4309258460998535, Training accuracy = 0.718750 : Test accuracy = 0.222222
Batch # =  300 : Loss :  1.2989866733551025, Training accuracy = 0.734375 : Test accuracy = 0.333333
Batch # =  350 : Loss :  1.1927057504653931, Training accuracy = 0.828125 : Test accuracy = 0.333333
Batch # =  400 : Loss :  0.9091487526893616, Training accuracy = 0.812500 : Test accuracy = 0.333333
Batch # =  450 : Loss :  0.8504770994186401, Training accuracy = 0.820312 : Test accuracy = 0.333333
Batch # =  500 : Loss :  0.7923119068145752, Training accuracy = 0.851562 : Test accuracy = 0.333333
Batch # =  550 : Loss :  0.7861653566360474, Training accuracy = 0.898438 : Test accuracy = 0.333333
Batch # =  600 : Loss :  0.9662376642227173, Training accuracy = 0.882812 : Test accuracy = 0.333333
Batch # =  650 : Loss :  0.6865573525428772, Training accuracy = 0.906250 : Test accuracy = 0.333333
Batch # =  700 : Loss :  0.5302461385726929, Training accuracy = 0.945312 : Test accuracy = 0.333333
Batch # =  750 : Loss :  0.4758228063583374, Training accuracy = 0.960938 : Test accuracy = 0.333333
Epoch  2/3 ########################################
Batch # =    0 : Loss :  0.4197838306427002, Training accuracy = 0.937500 : Test accuracy = 0.333333
Batch # =   50 : Loss :  0.3036743998527527, Training accuracy = 0.945312 : Test accuracy = 0.333333
Batch # =  100 : Loss :  0.4943648278713226, Training accuracy = 0.976562 : Test accuracy = 0.222222
Batch # =  150 : Loss :  0.5893290042877197, Training accuracy = 0.914062 : Test accuracy = 0.333333
Batch # =  200 : Loss :  0.5708660483360291, Training accuracy = 0.953125 : Test accuracy = 0.444444
Batch # =  250 : Loss :  0.4547967314720154, Training accuracy = 0.960938 : Test accuracy = 0.333333
Batch # =  300 : Loss :  0.3583762943744659, Training accuracy = 0.960938 : Test accuracy = 0.444444
Batch # =  350 : Loss :  0.4105512499809265, Training accuracy = 0.953125 : Test accuracy = 0.333333
Batch # =  400 : Loss :  0.3046374320983887, Training accuracy = 0.945312 : Test accuracy = 0.222222
Batch # =  450 : Loss :  0.3440867364406586, Training accuracy = 0.968750 : Test accuracy = 0.444444
Batch # =  500 : Loss :  0.4363264441490173, Training accuracy = 0.953125 : Test accuracy = 0.222222
Batch # =  550 : Loss :  0.4691902995109558, Training accuracy = 0.937500 : Test accuracy = 0.444444
Batch # =  600 : Loss :  0.3335230052471161, Training accuracy = 0.960938 : Test accuracy = 0.333333
Batch # =  650 : Loss :  0.3758492171764374, Training accuracy = 0.945312 : Test accuracy = 0.333333
Batch # =  700 : Loss :  0.4429911375045776, Training accuracy = 0.960938 : Test accuracy = 0.333333
Batch # =  750 : Loss :  0.2789595425128937, Training accuracy = 0.968750 : Test accuracy = 0.333333
Epoch  3/3 ########################################
Batch # =    0 : Loss :  0.2784363329410553, Training accuracy = 0.976562 : Test accuracy = 0.111111
Batch # =   50 : Loss :  0.3081202507019043, Training accuracy = 0.984375 : Test accuracy = 0.333333
Batch # =  100 : Loss :  0.2797819375991821, Training accuracy = 0.984375 : Test accuracy = 0.333333
Batch # =  150 : Loss :  0.3175572752952576, Training accuracy = 0.976562 : Test accuracy = 0.444444
Batch # =  200 : Loss :  0.2232826054096222, Training accuracy = 0.976562 : Test accuracy = 0.444444
Batch # =  250 : Loss :  0.3389220833778381, Training accuracy = 0.984375 : Test accuracy = 0.444444
Batch # =  300 : Loss :  0.3546279966831207, Training accuracy = 0.976562 : Test accuracy = 0.333333
Batch # =  350 : Loss :  0.2310389727354050, Training accuracy = 1.000000 : Test accuracy = 0.333333
Batch # =  400 : Loss :  0.2594821155071259, Training accuracy = 0.976562 : Test accuracy = 0.333333
Batch # =  450 : Loss :  0.1706273406744003, Training accuracy = 0.984375 : Test accuracy = 0.333333
Batch # =  500 : Loss :  0.2613034844398499, Training accuracy = 0.984375 : Test accuracy = 0.444444
Batch # =  550 : Loss :  0.2868270576000214, Training accuracy = 1.000000 : Test accuracy = 0.333333
Batch # =  600 : Loss :  0.2261587083339691, Training accuracy = 0.992188 : Test accuracy = 0.444444
Batch # =  650 : Loss :  0.3035414814949036, Training accuracy = 1.000000 : Test accuracy = 0.444444
Batch # =  700 : Loss :  0.2370609045028687, Training accuracy = 0.976562 : Test accuracy = 0.444444
Batch # =  750 : Loss :  0.2423712760210037, Training accuracy = 0.984375 : Test accuracy = 0.444444
Training accuracy = 0.984375 : Validation accuracy = 0.444444
Time usage : 0:00:56

The validation and test accuracy on the new test dataset are ~98% and ~44%, respectively. The deep learning model is not doing a good job in classifying the new dataset.

Let's plot the prediction.

In [80]:
#%% plot images - new test dataset
rows = 3
cols = rows
rc = rows*cols

images = newts_features

cls_true = newts_labels
cls_true = np.argmax(cls_true, 1)

cls_pred = y_pred_newts

plot_images(images, cls_true, rows, cols, cls_pred)

Calculating SoftMax Probabilities

In [81]:
# Print probabilities
for nimg in range(len(newts_labels)):
    print('Image # ',nimg)
    img = newts_features[nimg].reshape((imgsize, imgsize))
    plt.imshow(img, cmap='gray') 
    plt.axis('off')
    plt.show()
    print('True class : ',np.argmax(newts_labels[nimg]))
    print('p_class\tsoftmax_p')
    for nprob in range(len(softmax_pred_top5_newts[0][0])):
        print('{:>7}\t{:>9.5f}'.format(softmax_pred_top5_newts[1][nimg][nprob], softmax_pred_top5_newts[0][nimg][nprob]))
    print('#'*80)    
Image #  0
True class :  31
p_class	softmax_p
     31	  0.53355
     23	  0.27548
     21	  0.13178
     29	  0.02077
     19	  0.01745
################################################################################
Image #  1
True class :  20
p_class	softmax_p
     20	  0.51163
     28	  0.33623
     27	  0.14861
     11	  0.00262
     22	  0.00087
################################################################################
Image #  2
True class :  27
p_class	softmax_p
     12	  0.91318
     23	  0.02809
     38	  0.02593
     11	  0.02280
     21	  0.00287
################################################################################
Image #  3
True class :  40
p_class	softmax_p
     29	  0.72725
     24	  0.10533
     22	  0.04631
     28	  0.03642
     23	  0.03245
################################################################################
Image #  4
True class :  25
p_class	softmax_p
     25	  0.41014
     30	  0.35571
     11	  0.10470
     22	  0.05369
     28	  0.01360
################################################################################
Image #  5
True class :  0
p_class	softmax_p
     16	  0.39616
     40	  0.29540
      5	  0.11453
      3	  0.03620
     10	  0.03338
################################################################################
Image #  6
True class :  1
p_class	softmax_p
      7	  0.32374
      5	  0.20264
      8	  0.15464
      2	  0.13666
     10	  0.05983
################################################################################
Image #  7
True class :  14
p_class	softmax_p
     14	  0.92253
     38	  0.06964
      4	  0.00188
      0	  0.00174
     34	  0.00155
################################################################################
Image #  8
True class :  28
p_class	softmax_p
     23	  0.99964
     29	  0.00015
     28	  0.00012
     22	  0.00005
     19	  0.00003
################################################################################

Observations

Image Correct Label Predicted Labels (top 5) Discussion
31 [31, 23, 21, 29, 19] The classifier correctly classifed the image as class 31 with ~53.3% probability.
20 [20, 28, 27, 11, 22] The image is correctly classified as class 20 with ~51.1% confidence.
27 [12, 23, 38, 11, 21] As estimate earlier, this is a difficult image to classify. The classifier does not predict the class in top 5 predictions.
40 [29, 24, 22, 28, 23] As per earlier estimate, this is an easy image to classify. However, the image is incorrectly classified as 29. In the top 5 predictions, the classifier does not predict the correct class.
25 [25, 30, 11, 22, 28] The classifier correctly classifies the image with ~41% confidence.
0 [16, 40, 5, 3, 10] As per the earlier estimate, the image is not correctly classified. It is incorrectly classified as class 16.
1 [7, 5, 8, 2, 10] This prediction is strange. This is supposed to be easy classification. However, the classifier incorrectly predicts it as class 7. The classifier does not predict correct class in the top 5 predictions.
14 [14, 38, 4, 0, 34] As per the earlier estimate, this is supposed to be a difficult image to classify. However, the classifer predicts the correct class with ~92.2% confidence.
28 [23, 29, 28, 22, 19] As per the earlier estimate, this is supposed to be a difficult image to classify. Indeed, it is difficult to classify. The classifier does not predict it correctly. The correct label 28 appears in the top 5 predictions, however with mere ~0.01% confidence.

It is interesting to note that the model does not classify the new images on par with the testing dataset accuracy.


Question 1

Describe the techniques used to preprocess the data.

Answer:

The answer is provided in the section =>
Step 2: Preprocess and Generate New Data
Preprocess images (grayscaling and normalization)

Question 2

Describe how you set up the training, validation and testing data for your model. If you generated additional data, why?

Answer:

The answer is provided in the section =>
Step 2: Preprocess and Generate New Data
Generate Additional Dataset

Question 3

What does your final architecture look like? (Type of model, layers, sizes, connectivity, etc.)

Answer:

The answer is provided in the section =>
Step 3: Design and Test a CNN Model Architecture

Question 4

How did you train your model? (Type of optimizer, batch size, epochs, hyperparameters, etc.)

Answer:

The answer is provided in the section =>
Step 3: Design and Test a CNN Model Architecture
CNN Training

Question 5

What approach did you take in coming up with a solution to this problem?

Answer:

The CNN model is trained for different parameters such as learning_rate, epochs and batch_size. Parametic studies are conducted for each of these parameters and best ones are chosen to avoid overfitting or underfitting.


Step 3: Test a Model on New Images

Take several pictures of traffic signs that you find on the web or around you (at least five), and run them through your classifier on your computer to produce example results. The classifier might not recognize some local signs but it could prove interesting nonetheless.

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

The implemenation is described in
Step 4: Testing CNN Model on New Images

Question 6

Choose five candidate images of traffic signs and provide them in the report. Are there any particular qualities of the image(s) that might make classification difficult? It would be helpful to plot the images in the notebook.

Answer:

The answer is described in
Step 4: Testing CNN Model on New Images
Observations

Question 7

Is your model able to perform equally well on captured pictures or a live camera stream when compared to testing on the dataset?

Answer:

On live stream/captured images, the model will not perform on par with the testing dataset. The reason being the training/testing dataset is small. The model need to be build for a bigger datasets in different lighting conditions.

Question 8

Use the model's softmax probabilities to visualize the certainty of its predictions, tf.nn.top_k could prove helpful here. Which predictions is the model certain of? Uncertain? If the model was incorrect in its initial prediction, does the correct prediction appear in the top k? (k should be 5 at most)

Answer:

The implemenation is described in
Step 4: Testing CNN Model on New Images
Calculating SoftMax Probabilities

Question 9

If necessary, provide documentation for how an interface was built for your model to load and classify newly-acquired images.

Answer:

The pipelineis described in
Step 4: Testing CNN Model on New Images